一些好數字

08, 一個年份、一個月份、一個日子、一個時間

一百年前的奧運三問.. 在一百年後, 2008年8月8日晚上8時... 有了答案

51、21、28 ... 加起上來, 就是一百!

100 , 是一個圓滿的數字.

Design first

After joining the new company, the oracle always mentions "design first!".

At first, I do not really understand why proposing an implementation would get that message in return. Today, I read that again in a slack conversation. As an observer, my brain get flashed with a saying - "Design is problem setting, Planning is problem solving". Then, when that linked to the use cases that oracle keeps asking, I got something.

I'm really lucky to join this company, even though I'm not having much interactions with the oracle.

Things I learnt in the past 5 years

“It’s hard to teach a new dog old tricks.” - On many annual letters to shareholders written by the Oracle of Omaha

In the past 5 years, I worked with a team that have 2 product lines being alive while 5 others were dead. It is my pleasure to work with the team that let me learn and strengthen certain tricks. Old tricks take the past to learn and easy to forget, this post reminds my future self.

On risk taking

“Over the years, a number of very smart people have learned the hard way that a long stream of impressive numbers multiplied by a single zero always equals zero.” - Buffett

Startup could try to solve a problem with really new way of doings, only if the environment allows it. In the Valley, the legal and finance system allows a startup to take risky bet until the company gets enough eyeballs and/or moneys. This is not applicable to HK. We don’t have a court that knows technology, law that creates (or, at least try to create) a level playing field, money that supports risky bet, etc. So, being conservative sounds really legitimate here. No matter how much success you have achieved in the past, a single misstep could put you to the end. This also applies to engineering where new technology may not work as advertised, sometimes. Traditional technology with an active ecosystem implies battle scars on the face of others.

On decision making

Data driven is great, only when you can differentiate signal verses noise. Sometimes, a B2B SaaS could use data to drive decision, when the data looks consistent. Often, the sample size is too small to even make a reasonable guess. Leaders give strong opinion but weakly held. When we don’t have the data, we better admit that that is just a guess. (By the way, the infrastructure behinds data collection and data analysis is really hard to get right. I did that 3 times but not even come close to any.)

On product design

Minimum Viable Product means minimum, viable, and throw away. Even before you come up with the right question for the audience, how could you come up with an answer? This translates to “doing the right things is more important than doing things right”. Make tests to validate the question. MVP also applies to engineering. Leaky abstraction is bad but not even starting one sucks, whenever you find yourself repeat more than twice. That abstraction leads you to find your ultimate question. And, pre-mature optimization just wastes your time. Let it runs until it hits the wall.

On culture

“We shape our buildings and afterwards our buildings shape us.” - Churchill

Values should be deep-rooted while culture would be ever changing. Culture evolves around the values set by the team. Don’t be afraid to start with something small or vague. Once set, stick with it and shape it. Remember, making something up doesn’t mean doing it. Sometimes, peoples do make mistakes and toxic spreads in the speed of light, delay no more to apply the fix. Eventually, the team would be shaped by it. Engineering do also needs culture and value to scale. We shape our codebase and afterwards our codebase shape us. We eager to find something that can be repeated, nicely.

On multi-tasking

Some tasks can be run in parallel and some cannot. Most of the time, the best way is not to multi-task. Take pause, schedule the tasks using whatever you are comfortable with. Delegation enables true multi-tasking. In engineering, data processing shares this as well. Multi-processing sometimes does not work as expected.

On ownership

Ownership implies dedication. No-one can own everything since bandwidth is limited. In short term, sole ownership provides cost efficiency. In long term, that is a trap! Knowledge transfer is a daily routine that cannot be prevented. Being an owner of one thing makes you feel doing good, being an owner of multiple things makes you feel doing nothing. How could we take care of many things with the same level of dedication as one? Ownership transfer re-gains dedication.

On communication

Communication is king. Both written and verbal. Verbal is fast but lossy, written is persistent. They are not mutually exclusive and complement each other. In engineering, written is favored. Commit logs, documentations, source comments, discussion on issues. These help scaling and understanding. Not improving or ignoring it is fool.

On tooling

“Less is more.”

Having fewer things to care give you focus. Managed service like Heroku is always a good place to start with. Don’t try to build a custom PaaS when that is not your business. Though, the eager to reinvent a wheel should be respected, that gives you better understanding. Your business should already take your 80, the other 20 is better not to get more troubles.

On meeting

Meeting has many types and they all need an agenda. It is better used for discussion but brain-storming that could be done alone. Stick to the agenda, come to action items fast. And, treat synchronous meeting as a limited resource.

On tradeoff

“There are many ways to Rome.”

Most of the decisions have opportunity cost, taking one path may lose upsides of another. What really matter is whether that achieves the goal. And, making the tradeoff verbose provides better understanding. That could also gain support from the team.

Last but not least

Stay hungry. Stay foolish. Self-explanatory.

Thanks @anthonycyl for the edit.

​

Rate limiting Shopify API using Cuttle

During the development of a Shopify app, it is required to respect the API rate limit set by Shopify. Typically, we can use sleep() statement to make pause between API calls. This simple method works great until there are multiple processes that make API calls concurrently.

There are quite a number of ways to solve the problem.

1. Serialize all API calls into a single process, though not all business logics can work in this way.
2. Host a RPC server / use a task queue to make API calls. The RPC server / queue manager has to rate limit the API calls. [http://product.reverb.com/2015/03/07/shopify-rate-limits-sidekiq-and-you/]
3. Centralize all API calls with a HTTP proxy where the proxy performs rate limiting.

Personally, I think the RPC server / task queue option is quite heavy weighted since that requires:

* A RPC / task framework, and
* A RPC server / task queue, and
* A rate limit system built around the RPC server / task queue.

In contrast, the HTTP proxy option only requires a HTTP proxy server plus a HTTP client. And, HTTP is well supported in many programming languages and systems. It sounds as a great starting point.

(BTW, HTTP can be considered as the underlying protocol of a RPC system.)

With the HTTP proxy option, there are quite a few options to get started.

1. Use Nginx reverse proxy to wrap the API, use its limit module to perform simple rate limit or write a Lua/JS plugin for more sophisticated control. [http://codetunes.com/2011/outbound-api-rate-limits-the-nginx-way/]
2. Use Squid forward proxy to perform simple rate limit by client info (e.g. IP address).

At the first glance, the Nginx reverse proxy option looks superior since we can have sophisticated rate limit control deployed. Though, using such approach would need to use the Nginx wrapped URL of Shopify API. Or, we have to modify DNS/host configuration to route the traffic.

Personally, I am not comfortable in modifying the URL to Shopify API since that may prevent a smooth upgrade of the Shopify API client in the future. For the DNS option, shall I modify the DNS config once per a new Shopify store install the app?

(We may also route all traffic to the default virtual host of Nginx and use Lua/JS plugin for the host routing. This does not require URL wrapping or DNS configuration. Though, I personally think this is kinda abusing Nginx.)

So, reverse proxy may not be a good way to go. Let's come to the forward proxy option. In this case, we do not need to do anything on the URL to Shopify API and just let the traffic goes through the proxy by configuring the HTTP client. A forward proxy with rate limit control sounds like a good way to go.

Here, we come to Cuttle proxy. [http://github.com/mrkschan/cuttle]

Cuttle proxy is a HTTP forward proxy solely designed for outbound traffic rate limit using goroutine. It would provide a set of rate limit controls for different scenarios. In case of Shopify API, we can use the following Cuttle settings to perform rate limiting.

addr: :3128
zones:
  - host: "*.myshopify.com"
    shared: false
    control: rps
    rate: 2
  - host: "*"
    shared: true
    control: noop

Then, set the HTTP proxy of the Shopify API client like below to route API calls through Cuttle.

# apiclient.py
import shopify

shop_url = 'https://{}:{}@{}/admin'.format(API_KEY, PASSWORD, SHOPIFY_DOMAIN)
shopify.ShopifyResource.set_site(shop_url)

print json.dumps(shopify.Shop.current().to_dict())

# Run
HTTPS_PROXY=127.0.0.1:3128 python apiclient.py

As long as all API clients are configured to use Cuttle, API calls will be rate limited at 2 requests per second per Shopify store. So, the rate limit bucket would rarely go empty.

Note: It is up to you to set the rate of API calls in Cuttle, using 3 requests per second per store would be another great option. You will receive HTTP 429 sent by Shopify roughly after 120 continouos API calls to the same store over 40 seconds.

Note: API calls will be forwarded by Cuttle using the first come first serve manner. If the concurrency level of API calls to the same Shopify store is high, some API calls will wait for a significant amount of time instead of receiving HTTP 429 sent by Shopify immediately. Remember to set a reasonable HTTP timeout in that case.

(FYI, the Shopify API rate limit only favors high concurrency level for a short duration. If you really need that in your case, Cuttle would not be a good option.)

Scope finding in a source file

This post is going to discuss an issue I met when building a text editor plugin that tries to find the class/function scope which the current line on the editor belongs to (http://atom.io/packages/ctags-status). The problem I met can be broken into two parts: (i) Given a set of ranges that may be overlapping on a one dimension plane, find the ranges that cover a point on the plane. (ii) Given a set of overlapping ranges, get the topmost range where the height of ranges follows the ascending order of the starting point of all ranges (the higher in the stack, the later in the sequence). Note, the issue is not a hard problem. This post documents how I encounter and work on the problem.

So, here is the story.

When I build the early version of the plugin, I want to ship it as soon as possible and see if it is downloaded by anyone (Atom editor does not expose plugin usage data to its author yet, so the only number I have is downloads). Thus, there was not much thought process in those days.

The early implementation models each scope as a range with start and end line. To find the scope that the current line belongs to, the problem becomes a range search problem. Ranges would be overlapping when there is nested scope. In that case, the start and end lines of the inner scope would always be enclosed by those of the outer scopes. So, I can sort all scopes by their start line in ascending order, and the innerest scope on the current line would be the last one in the sequence that its line range encloses the current line. This is a O(N log N) preprocessing + O(N) lookup. I was happy with it.

So far so good?

The issue was not surfaced until I used the plugin to browse a long source file that has dozens of functions (yup, shouldn't the file be split for readability?). When I kept moving down the cursor for a while, its movement was no longer smooth. The issue was that the plugin needs to find the scope upon each cursor line change. When I fired up the profiler, I found 300 - 400ms were spent on scope finding when there were dozens of continuous cursor line changes. I was not sure whether the plugin was really the cause of the UX problem but it is the one that took most of the processing time. So, time for optimization!

Since this is a range search problem, KD tree, segment tree, and interval tree quickly came to my mind. There are several factors to consider in picking a solution: (i) availability of existing implementation (I don't like reinventing without enhancement), (ii) speed of insert / delete / update (when a source file is edited, there is a high chance that scopes are moved), and (iii) lookup speed of course. When I was still deciding which search tree best fits the issue, I raised a questions to myself. Why don't simply hash the scope(s) on each line? A simple hash with a stack in each bucket is a good fit because:

(i) I just need JavaScript object (hash) and array (stack) to build it.
(ii) A typical source file has less than thousand of lines with a dozen of scopes. The worst case is having thousands of pointers (lines * scopes) referring to a dozen of strings (scope names). That should not take a lot of spaces.
(iii) A file edit would introduce quite a lot of scope movements (e.g. insert a new line at top of the file pushes all scopes down). Maintaining a data structure via insert / update / delete is like rebuilding it in the worst case. Building a big hash takes O(NL), number of scopes * number of lines in the file (which is several thousands of iterations). The hash building process is offline and I don't expect it would take long, so I am happy with that.
(iv) O(1) lookup, the best that I can get.

As a result, the plugin is using a hash for scope finding.

Python descriptor, Django CharField with encryption

This post is part of the pyfun series, I will try to *log* some of the features that I think they make Python funny :)

One of the most recent topics in my reading list is Python descriptor.

An object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol. Those methods are __get__(), __set__(), and __delete__(). If any of those methods are defined for an object, it is said to be a descriptor. - http://docs.python.org/2/howto/descriptor.html

When I finish the howto on python.org, I don't really understand what is it and thus my read-later list kept expanding with a lot of related articles; until I came across this post (If you don't know what is Python descriptor, I recommend you to read the post first since I'm not here to re-post the details with my poor English).

The purpose of this post is to extend the recommended reading to provide another example use of Python descriptor - a encryption/decryption wrapper of a Django `CharField`.

One of the major purpose to implement a Python descriptor is to provide getter and setter to attributes. In some traditional programming languages, we have to implement/generate a set of getter and setter to protect the read/write access of attributes. Or, we can use a generic attribute class that has the protection but the access of the attributes looks like `object.attribute.get()` and `object.attribute.set(xxx)`. Python descriptor solves both of the mentioned problems.

To encrypt/decrypt a `CharField`, it is obvious to override its `get()`/`set()` functions. We can simply do so by extending the `CharField` just like this snippets. However, I would like to demonstrate the use of Python descriptor (yep, I'm abusing it here).

At first, we need the descriptor with encryption and decryption. The cipher we use here is a simple 32-bytes XOR without padding (which is simply uesless in most of the cases).

class EncryptedAttr(object):
    '''Descriptor that encrypt content on write, decrypt on read'''
    def __init__(self, attr, secret_key):
        self.attr = attr
        self.key = secret_key

    def encrypt(self, v):
        '''A simple XOR chiper'''
        return ''.join(chr(ord(a) ^ ord(b)) for (a, b) in zip(self.key, v))

    def decrypt(self, v):
        '''A simple XOR chiper'''
        return ''.join(chr(ord(a) ^ ord(b)) for (a, b) in zip(self.key, v))

    def __get__(self, obj, klass):
        '''Get `attr` from owner, and decrypt it'''
        cipher_text = getattr(obj, self.attr, None)
        if not cipher_text:
            return ''

        return self.decrypt(cipher_text)

    def __set__(self, obj, value):
        '''Encrypt value, and set to owner via `attr`'''
        if not value:
            setattr(obj, self.attr, '')
            return

        cipher_text = self.encrypt(value)
        setattr(obj, self.attr, cipher_text)

The descriptor requires a Django model attribute name and a secret key in its constructor. The attribute name is used to look up the wrapped attribute of the Django model in its `__get__()` and `__set__()` functions. To use it, we just assign it as an attribute to the model class.

class Secret(models.Model):
    wrapped = models.CharField(max_length=32)
    content = EncryptedAttr('wrapped', 'This is the 32-bytes secret key.')


# Let's make a secret
payload = 'The secret must be 32-bytes long'  # Because we use a 32-bytes XOR
s = Secret()
s.content = payload

s.wrapped
>>> '32-bytes blah blah blah blah ...'

s.content
>>> 'The secret must be 32-bytes long'

In this example, the CharField `wrapped` attribute is not expected to be accessed directly. When we assign plain text to `content`, the plain text is encrypted and stored to `wrapped`. The `content` attribute does not hold anything at all. On the other hand, when we read from `content` attribute, it actually decrypts the cipher text from `wrapped`.

You may get the sample Django project to play around at https://github.com/mrkschan/encrypted-field.

Partial function call

This post is part of the pyfun series, I will try to *log* some of the features that I think they make Python funny :)

Again, I was reading the Scala tutorial and find that she has built-in support of partial function call (see http://twitter.github.io/scala_school/basics.html#functions). This reminded me that Python does also has functools.partial(), which can be used as function shortcuts.

Let's see this example of Django.

# Let's have a Coupon that can either be fixed amount discount or percentage off
# But, we don't want to have model inheritance and table join to get the data
class Coupon(models.Model):
    code = models.CharField(max_length=8)
    type = models.CharField(max_length=11, choices=['fixedamount', 'percentage'])
    amount = models.DecimalField(max_digits=8, decimal_places=2)
    currency = models.CharField(max_length=3, choices=['USD', 'CAD'], default='')


# To create a Coupon based on certain conditions, you can have this
kwargs = {'code': code}

if condition_a:
    kwargs.update({'type': 'fixedamount', 'currency': 'USD'})
elif condition_b:
    kwargs.update({'type': 'percentage'})

kwargs.update({'amount': x if condition_c else y})
coupon = Coupon.objects.create(**kwargs)


# Or with functools.partial()
FixedamountCoupon = functools.partial(Coupon.objects.create, type='fixedamount')
PercentageCoupon = functools.partial(Coupon.objects.create, type='percentage')

if condition_a:
    coupon = functools.partial(FixedamountCoupon, code=code, currency='USD')
elif condition_b:
    coupon = functools.partial(PercentageCoupon, code=code)

coupon = functools.partial(coupon, amount=x if condition_c else y)
coupon = coupon()

Yes, we just shortcuted two types of Coupon using functools.partial(), created "sub-class" of Coupon. Furthermore, if the underlying function accepts positional arguments, we can also shortcut those arguments.

Clean code FTW.

Scala Traits in Python?

Sometimes, I would question myself, am I qualify as a seasoned Python engineer? The answer is I'm still a junior. The next question is, how to qualify as a seasoned one?

I have two things in mind that may be part of the answer to the second question:

1. Know when to use certain Python features.
2. Know what kind of library or framework is suitable for particular task.

In this pyfun series, I will try to *log* some of the features that I think they make Python funny :)

Let's get back to the primary subject of this post. Scala, which compiles to JAVA bytecode and runs on JVM, is one of the latest big hit. So, I'm going through some tutorials of it to check out her magics. One of the topic is Traits.

Traits, is similar to Java interface. When I read some samples about it, I wonder how can it be done in Python. From stackoverflow, http://stackoverflow.com/q/6240118, there is a suggestion that to use Python class to simulate the thing. And, starting from that idea, I wonder can we do that in runtime instead of just static definition. Here, we come to Python type().

Most of the time, type() can be used as Enum in C (see http://stackoverflow.com/a/1695250), or dynamic objects like those in Javascript (https://gist.github.com/mrkschan/6936112). It is because type() is essentially a dynamic form of the class statement (http://docs.python.org/2/library/functions.html#type). In other words, we can use type() to create Python class that simulates Traits (JAVA interface) at runtime.

Here is an example.

# See original Scala version at http://twitter.github.io/scala_school/basics.html#trait
Car = type('Car', (object,), {'brand': ''})
Shiny = type('Shiny', (object,), {'refraction': 0})

BMW = type('BMW', (Car, Shiny,), {'brand': 'BMW', 'refraction': 100})
my_bmw = BMW()

print my_bmw.brand, my_bmw.refraction

# We can have constructor as well
def bmw_init(self, refraction):
    self.refraction = refraction

BMW = type('BMW', (Car, Shiny,), {'brand': 'BMW', '__init__': bmw_init})
c1, c2 = BMW(10), BMW(100)
print c1.refraction, c2.refraction

As we can see, we can use type() to create a set of "interfaces". And, use type() to create class that implement the "interface". These "interfaces" can have its name changed according to runtime conditions. type() is Python magic and it's fun :)

However, I did not come up with a use case for runtime defined Traits yet :P

Bypassing (Great) firewall to access GitHub / BitBucket via SSH Tunnel

Sometimes, you may be blocked by a firewall and cannot access GitHub / BitBucket. In this post, the steps to bypass the firewall using SSH tunnel is documented.


Step 1 - Setup the tunnel
----------------------------------------

Assuming you use SSH to perform git operations (git clone, fetch, pull, merge, etc.), you should find a SSH URL like: git@github.com:example/example.git or git@bitbucket.org:example/example.git. In order to access the blocked SSH hosts, we have to SSH tunnel to forward the requests. To do so, use the following commands to create a tunnel (assume you have a SSH host that can be accessed).

ssh -C -L 8022:github.com:22 example@example.com  # Establish a tunnel to github.com, SSH requests to local port 8022 are forwarded to github.com:22.

ssh -C -L 8122:bitbucket.org:22 example@example.com  # Establish a tunnel to bitbucket.org, SSH requests to local port 8122 are forwarded to bitbucket.org:22.

Of course, you can combine the two to have one SSH session only.

ssh -C -L 8022:github.com:22 -L 8122:bitbucket.org:22 example@example.com


Step 2 - Config SSH client
------------------------------------------

After you have your tunnels, you can then configure your SSH client to redirect SSH requests to your tunnels. Put the following lines in your ~/.ssh/config file.

Host github.com
HostName 127.0.0.1
Port 8022

Host bitbucket.org
HostName 127.0.0.1
Port 8122


Afterwards, feel free to use git clone git@github.com:example/example.git or git clone git@bitbucket.org:example/example.git (you can also use git fetch, pull, merge, etc.). All requests will be passing through your SSH tunnel.

中產工種被誰取代

昨天在看一篇評論，說美國總統做什麼都改變不了大部分中產階層的命運。理據如下：

1. 大學生普及化，一般水平的人力資源充足，而需求卻沒有相對增加，使一般中產階級工種薪酬下降，而且惡性循環不斷。

2. 軟件業發達，企業軟件取蒂中產階級工種人手(如中級管理人員)，高層工作效率因軟件發達而不斷上升，促使人手需求不斷下降。

那麼，要擺脫那惡性循環，除了是優秀人才，或是專業工作從業者，或是成功startup的核心團員。還有什麼正常途徑？扎鐵吧！

作為軟件行業從業員, 又如何擺脫厄運?

Laptop火牛點解咁大隻

Laptop本來的設計精要在於移動性，

但係好多laptop的火牛，都差不多是一罐可樂咁重(最近習慣了用可樂做單位）。帶laptop出街 ，本來是1.5kg卻變了2kg，實在無耐。

點解火牛要咁大隻，唔可以好似Apple果D維他奶size？

我有谂過買多一隻牛，公司一隻屋企一隻，咁就少左個藉口唔帶Laptop，但又不想為一部舊laptop投入過多金錢… 你話人係不係總要犯賤?又要解決問題又不肯付出：P 還是多花一兩舊水，把懶惰的藉口抹去吧啦

SEO Tips: Abuse Github

NOTE: I'm not a SEO guy. If this post used wrong wordings, sorry for that.  And, I'm not a native English speaker, don't blame my English writing :P And, I don't know whether someone has shared something similar :)

I have several pet projects hosted on github. And, I observed that Google give pretty high ranking to github repository. If a github repository has a homepage, that homepage will get benefits as well.

Thus, I carried out an experiment at https://github.com/mrkschan/github-seo-effect. I used the search terms "Github SEO effect" as the repository name and set the homepage that links to my blog post at http://mrkschan.blogspot.hk/2013/03/github-seo-effect.html

The result is pretty amazing. Here is a Google search result before I carried out the experiment (I was using Chromium private browsing, with links set to use "gl=us").

My blog post gets nowhere on the search result.

After a few days that I set the homepage to the blog post, the result becomes the following screencap.

My repository goes to the top of the search result. And, the blog post is on the first page of the Google Search :)

BTW, shall we abuse Github for SEO purpose? Try your search here: http://www.google.com/search?gl=us&q=github+seo+effect

Github SEO effect

This is a blog post to experiment the hypothesis:

Google would give github repo pretty high ranking.
If a repo's README has a link to a page outside github, that page gets SEO bonus.edit: 20130308 If a repo's homepage has set to a web page, that page would get SEO bonus from Google (I didn't carry out experiment on the effect of the README yet). And, if the page links back to the github repo, that's a plus.

Search google with the query "Github SEO effect" :)

Link: https://github.com/mrkschan/github-seo-effect

Result: http://www.google.com/search?gl=us&q=github+seo+effect

信念, Faith

男人問女人: "你有什麼信念嗎?"

女人問: "為什麼問這個問題?"

男人說: "別人說人不能沒有信念."

女人答: "我相信會找到一個愛我的人."

女人問: "那你呢?"

男人說: "我不知道."

朋友們, 你的信念是什麼?


Faith - (from thefreedictionary.com)

1. Confident belief in the truth, value, or trustworthiness of a person, idea, or thing.

2. Belief that does not rest on logical proof or material evidence.

信念 - (from 中華民國教育部 - 重編國語辭典修訂本)
信仰不疑的意念.


我的信念? 可能是建立一個舒適的家 :)

很久沒寫

最近寫的, 不是生活相關的, 就是技術相關的, 缺少了思考的東西. 人像懶了, 笨了.

原因? 曾經想過會不會是生活太匆忙而給自己藉口不去思考東西/反思. 又會不會是習慣了 twitter 文化, 一切從簡? 那今天又為什麼在寫這篇廢話呢? 不是在 twitter 上寫句: o, i'm so lazy. 就行了嗎?

事原: 有位家人最近離開了這個世界, 我因而接觸到佛學的地藏王菩薩經. 裡面, 提到了無間地獄. 無間地獄在電影無間道被提及過: "明明我已晝夜無間踏盡面前路, 夢想中的彼岸為何還未到". 再加上最近有點迷失(even after i read the book "Life is what you make it" twice), 對無間地獄提起了點興趣. 剛坐車回家的時候, 在想如何去認識無間地獄. 忽然, 想起了在大學的時候, 學習道家文化時在這個 blog 寫了點東西. 到家了, 便走上這裡, 找找以前寫的東西. 期間, 我在看自己以前寫的東西. 整個過程很有趣. 因為我在嘗試認識從前的我. 看了, 發覺對從前的我很陌生, 而且這個過程很有趣 :)

為此有感而發, 在twitter文化驅使下寫了句 "regrets should be prevented, but reviewed is encouraged. http://mrkschan.blogspot.hk/2006/09/blog-post.html … my 1st blog :)". review的就是要多認識自己, regret的就是我的靈魂好像有一年時間消失了一樣.

想了一會, 我不想以後再後悔. 花少少時間, 寫一寫, 裝個讀書人.

How to use Shovel with Django

Here is my attempt to use Shovel (Rake, for Python) in Django.

Shovel can be used as an alternative to Django built-in Command. The reasons to use Shovel instead of the Command are:

1. "Rake, for Python" sounds cool.
2. Shovel is light-weight.
3. I can find all tasks in one place (the shovel/ directory).
4. I hate writing a list of make_option.
5. I hate inheriting LabelCommand, BaseCommand, NoArgsCommand, etc.

Okay, you get it... The reasons to use Shovel are actually: (i) "Rake, for Python" sounds cool, and (ii) I am lazy.

I didn't use the term "replacement for Django Command" because we cannot simply invoke Shovel task in the Django codebase using django.core.management.call_command(). (But we can have work-around, like making a Command to proxy the call)

Let's get back to my attempt to mix the two. I placed the shovel/ folder inside the Django project.

djangoproject/
  - manage.py
  - settings.py
  - urls.py
  - shovel/ 

And I setup the Django environment at the top of the Shovel task file.

try: # Load the django environment
    import imp
    import os
    import sys

    me = os.path.abspath(os.path.dirname(__file__))
    module_info = imp.find_module('context', [me])
    imp.load_module('context', *module_info)
except:
    print >> sys.stderr, 'Cannot setup Python environment from context.py'

At last, the setup injects a directory to Python sys.path and use Django function to load the settings.py.

def setup_django_env(path):
    from django.core.management import setup_environ
    try:
        module_info = imp.find_module('settings', [path])
        settings = imp.load_module('settings', *module_info)

        setup_environ(settings)
    except ImportError:
        msg = "Error: Can't find 'settings.py' in configured PYTHON_PATH", path
        print >> sys.stderr, msg
        sys.exit(1)

# assume shovel/ directory is placed at the same level of settings.py
setup_django_env(os.getcwd())

This is a common technique to integrate Django with other Python gears as well. By applying this technique, I can freely use any Django functions / models in Shovel.

NOTE 1: The name of Shovel task file cannot collide with any module/app name of the Django project.

NOTE 2: We can place the shovel/ folder outside the Django project but we have to use proper Python path and import statement (e.g. use `from themix.things.models import Thing` instead of `from things.models import Thing`).

My attempt can be found at - https://github.com/mrkschan/shovel-django-mix.

git-fix-whitespace series 1: Knowing about `git diff -p`

In the last post, the first requirement of the project (https://github.com/mrkschan/git-fix-whitespace) has been settled. The next requirement of this project is to read the git-diff patch in order to find any line changes that violate the whitespace rules specified in git config.

In a typical git-diff patch (see below), there are a few major parts.

1. Line 1 provides metadata about the modified file. It gives the file path to the file, rooted at the git repository.
2. Line 2 provides metadata about the git index and the file object discretionary access control list.
3. Line 3 and 4 provides metadata about which file path is old and which is new.
4. Line 5, 14, and 24 are metadata that tells which part of the file is modified. Let's take an example to explain this - "@@ -33,8 +33,7 @@ def sanitize_diff(git_diff):". "-33,8" tells that there is a diff hunk with 8 lines starts at line 33 of the old version of the file. "+33,7" tells that there is a diff hunk with 7 lines starts at line 33 of the new version of the file. As a result, the new version of the file gets one line fewer than the old version.
5. The rest of the patch is the content in the modified file. Those lines are either prefixed by ' ', '-', or '+'. ' ' means no modification, '-' means removed line, and '+' means added line. Note, there is no line replacement since it is represented by '-' lines followed by '+' lines (see line 28-31).

After knowing the structure of a git-diff patch, the next step is to read and write the modified file.

git-fix-whitespace series 0: GitPython vs libgit2

This is the first post of the git-fix-whitespace series. In this series, I will put some notes about working on the project - https://github.com/mrkschan/git-fix-whitespace. (NOTE: This series is a by-product of the git-fix-whitespace project, since my blog need some updates :P)

First of all, let me introduce the rationale of working on this git-fix-whitespace project. Most of the time, as a Python developer, I hate tab indentation and trailing whitespaces (read pep8). I know there are existing tools to "proof-read" a file using certain whitespacing rules. However, I insist to create my own tool to achieve the goal. Reason: I just wanna have a pet project that I LOVE to keep working on.

The very first requirement of this project is to support the configuration directives of git (see `man git-config` and look for `core.whitespace`). Hence, I need to find a tool that can read the git configuration files (both the user level git config file ~/.gitconfig and repository level git config file /.git/config). By using Google, I got GitPython and libgit2/pygit2. At first, I try to read the libgit2 python binding to see if I can read whitespace configuration by simple api calls. But it seems that it did not support that yet (as of 2012-Jan-15). Then I move on to GitPython. Gotcha! There's simple api call to read the core.whitespace configurations :) As a result, git-fix-whitespace got a dependency on GitPython at the moment.

話說的好滑腸粉

聽阿媽話說, 97年果時有位茶樓點心大廚, 因為人工高被人炒魷, 結果到了沙田博康村的大排檔拉腸粉. 今日, 本來是要跟阿媽行山去川龍飲茶的日子, 但因為唔多想坐車入荃灣咁遠, 結果是從獅子山行入沙田... 朝早7:30出門口, 9:30到了博康村. 剛剛好趕得上腸粉時段 (NOTE: 果間野 d 即拉腸粉, 只限每天早上十時前供應).

這即拉腸粉的滑口程度, 絕對比米豬蓮添女子運的腸粉好得多... And, 添女子運沒有花生醬及甜醬供應....

Stats for Firefox 4 Release Party on May 14

As you may know, we're going to have a Firefox 4 Release Party this Saturday in Hong Kong!! (see http://opensource.hk/node/666 :)

I would like to share with you some interesting stats we collected from the registration :)

At first, let's see who's coming :)



Most of us are user of Firefox :) You can find contributor as well :)

Then, which OS we use most???



Feeling sorry to "vista" ... what is it -.-? Anyway, what are we going to do this Saturday ?? hehe... see below :)