Thursday, December 29, 2011
And worse - the usual LAMP stack is terrible at handling anything that's actually asynchronous in nature, like WebSocket or Comet. It can also be a pretty big security hole as well, see Slowloris.
And yet, you have to read lengthy manuals and fiddle with lengthy config files to get anything done with the traditional web servers - instead of getting shit done, like, right now. Seriously, this is 2011. You're doing more for less if you're sticking to the 2000 ways.
Let's try another way.
So, let's say you have a usual Ubuntu, or Debian, or the more trendy Mint machine. Try this in your home directory:
$ sudo apt-get install libevent-2.0 libevent-dev python-dev python-virtualenv $ virtualenv webservice $ . webservice/bin/activate $ pip install gevent gunicornWell, congratulations! You've already got a self-contained web server package installed with the above four lines of shell commands! If your Linux machine has libevent and virtualenv installed already, you can even cut out the first line.
So, 3 lines of shell code to get the infrastructure ready.
The Apache guy is.. eh.. still looking through the manual pages for how to set up virtual hosts. We'll get back to him later, after the late-night news, supper, and.. watching grass grow.
Now, for the web server's content, we'll try a Hello World first. It'll be a Python script for now - we'll make it more like a real web server (i.e. serves your .html, .js, .css and runs scripts and frameworks just like your Apache, nginx, etc.) in another post.
$ cd webservice $ cat > application.py #!/usr/bin/env python def application(environ, start_response): start_response("200 OK", [("Content-type", "text/plain")]) return [ "Hello World!" ]Just in case you're not familiar with UNIX, press Ctrl-D after the last line to save the file.
So, 3 lines of Python code to make the server serve.. something.
Finally.. we start the web server:
$ gunicorn -w8 -k gevent --keep-alive 60 application:applicationAnd.. it's alive! It's running at port 8000 by default. You can see it in http://localhost:8000/
The performance isn't shabby as well, considering the web server part of it (i.e. gevent.pywsgi, which handles the underlying HTTP protocol) is written in 100% Python:
$ ab -c 32 -n 12000 http://localhost:8000/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 1200 requests Completed 2400 requests Completed 3600 requests Completed 4800 requests Completed 6000 requests Completed 7200 requests Completed 8400 requests Completed 9600 requests Completed 10800 requests Completed 12000 requests Finished 12000 requests Server Software: gunicorn/0.13.4 Server Hostname: localhost Server Port: 8000 Document Path: / Document Length: 12 bytes Concurrency Level: 32 Time taken for tests: 1.182 seconds Complete requests: 12000 Failed requests: 0 Write errors: 0 Total transferred: 1644000 bytes HTML transferred: 144000 bytes Requests per second: 10148.61 [#/sec] (mean) Time per request: 3.153 [ms] (mean) Time per request: 0.099 [ms] (mean, across all concurrent requests) Transfer rate: 1357.77 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.2 0 4 Processing: 0 3 1.9 2 22 Waiting: 0 3 1.9 2 22 Total: 1 3 2.0 3 23 Percentage of the requests served within a certain time (ms) 50% 3 66% 3 75% 4 80% 4 90% 5 95% 6 98% 8 99% 11 100% 23 (longest request)For comparison, Apache 2.2 running on the same machine, serving a static file does 12k requests per second. And you can pretty much forget about playing with Comets in it without a lot of configurations and modifications.
Monday, December 26, 2011
Monday, November 28, 2011
The Easy Way Out
Salesmen and Bean Counters
Saturday, June 11, 2011
“We’re doing our next mobile product in HTML5.”
I’ve been having a lot of this kind of conversations lately - partly because I’m doing an HTML5 app myself. But like “AJAX” back in 2005, the term “HTML5” is never clearly defined and is quite easy to throw around without considering the actual technical benefits, the trade offs, and the implications in the operations side. If you’re working in a company managed by buzzwords of the day, then Dilbert may very well be sitting by the next cubicle - if you’re lucky.
Two Divergent Roads
The more intuitive, and technically simpler path is to use the web browser - either embedded to your app (e.g. PhoneGap) or, the app is supposed to be opened from a bookmark icon on your mobile phone (e.g. Financial Times for iPhone and iPad). The usual benefits of this path include being able to reuse existing web design and development talents, products are highly and trivially portable, are easy to debug and fix (e.g. via weinre), and thus, fast iteration. Trade offs are performance and features - features is less of a hassle if you use an embedding architecture like PhoneGap. Performance is the #1 problem for this path.
Node.js for Client Side
Back to the Browser
Two UI Libraries
jQuery Mobile is a project founded late last year so it’s relatively new. It is quite obviously immature to anyone having actually tested it - persistent navigation bars are broken, page switching is noticeably slower than native, no way to incrementally add to lists without refreshing the whole page, high CPU usage when tested on desktop platforms (as of alpha 4), etc. I chose it for my project for its relative simplicity (i.e. hackability), the MIT license, and because it’s based on jQuery - every respectable web developer knows jQuery so it’s easy to train someone in jQuery Mobile.
Sencha Touch is, purportedly, much more mature and is faster. I have an instinctive abhorrence towards things that comes with a higher complexity, because I always have a feeling that there are gonna be a lot of features that I wouldn’t use and end up as dead weights pulling down the overall performance. I may be wrong though - the top featured mobile app on PhoneGap’s apps page is IGN Dominate - it’s extremely fast and it’s based on Sencha Touch - although I’m sure they’ve put in a lot of their own optimizations to do that.
Debugging and Making Changes
The best solution right now is weinre. It is buggy and slow but it works - with that you can literally debug your mobile app’s UI from Web Inspector on your computer. It is based off WebKit’s Web Inspector with its debugger back end taken out and replaced with a remote back end. I’ve done some investigations into Web Inspector’s code myself two or three weekends ago and found that it’s not really that hard to turn it into a remote debugger. weinre’s development may speed up in the coming months or someone may make a better project replacing it - we’ll see.
This will be the actual big thing in mobile app development for the coming years, because it solves tons of problems in the 80% part of the app developer’s workload. Want to change your UI design in Objective-C? Edit, recompile, run - repeat until you get satisfactory results. This can easily take a whole day if the recompile step is long. Doing that same thing with HTML5? Edit a few CSS attributes in weinre and experiment away - you don’t even need to shut down the app. Yes you can debug an HTML5 mobile app, to some extent, by running it in a desktop browser. But believe me - your app will have a bunch of bugs that happens only on the mobile device, so weinre is absolutely needed.
Unfortunately, people usually tout their toolkits and their features and seldom glorify useful debuggers, so I guess it won’t get talked a lot even though it’ll actually be the thing that mobile HTML5 developers use the most.
The Present State
- The JS engine approach makes sense for making games. I have reservations for implementing a full UI toolkit on that due to the complexities involved in the interfacing layer - but, we’ll see.
- Full-stack HTML5 mobile app is slow and buggy, but can be mitigated with custom optimizations. Not many people have managed to do that but it's possible.
- The real benefit of doing HTML5 mobile apps is the possibility of debugging and changing it online. Native app developers will have a hard time matching that kind of development speed advantage - once the team has problem #3 resolved.
- Oh yes, porting is easy as well. But I assume everyone gets that automatically.
Thursday, June 2, 2011
Saturday, May 28, 2011
- The current approach of embedding scripts for all browser support in libraries like jQuery, and embedding all available widgets in libraries like jQuery UI and jQuery Mobile - is completely wrong. The extra parsing and code complexity may not matter for Core i7 desktops, but it really hurts performance in mobile devices.
- If the uncompressed size of the .js file of the UI framework for your upcoming mobile app is 1.5MB+ ... you're doing something very wrong.
- Narrative JS. Can be used to solve a lot of problems if it works well. I should find some time to investigate this thing.
- I need to do more screencasts like AFeature's, for those "micro" open source projects (like bson and bson-objc) that I do in weekends - just to brush up my presentation skills.
- The current way people do Comet and WebSockets haven't accounted for unreliability at the physical layer on mobile devices (aka. AT&T).
- Comet on mobile devices is also not a very good practical idea because (from our experiments), mobile browsers tend to minimize active TCP connections - s.t. even though you can sort of keep the receiving connection "persistent", the send will not. Which means, the send part of your Comet session will have high latency.
Sunday, May 22, 2011
- Mobile applications will increasingly be done in HTML5 and wrapped inside something like PhoneGap.
- Co-routine I/O libraries like gevent and eventlet, and the whole approach in general irrespective of languages, will take off. Someone will implement an Actor model on top of them to make them work nicely across processes. (note: Pykka doesn't work across processes, yet)
- NowJS will be a flop.
- ChromeOS will be a flop - for now. But Google has the money and patience to iterate.
- WebSockets and WebGL will remain experimental.
- The general direction for software that'll take off are things that are lean and mean - in terms of performance, code size, ease of development, etc.
- Everybody that matters will will say UX instead of UI - which means just a good UX will no longer confer significant competitive advantage in general.
- The HK government will keep pushing meaningless things like "the wine industry" where a single tech company will easily trump the whole industry in terms of revenue - but they'll keep talking the talk on tech.
- HK university graduates still don't know their mainland counterparts from similarly high-ranked universities are already getting a higher salary and maybe 2x to 3x their real purchasing power.
- The world will not end tonight, doomsday preachers are nuts.
Saturday, May 21, 2011
The code has been in use for several months without issue and has been working fine all this time in all the browsers I’ve tried it in. For some reason in this one particular scenario though, Safari was completely ignoring the attempts to set the innerHTML of the node. Setting the innerHTML and then on the following line attempting to read it was also giving an empty response. For example:
text = "foo";
node.innerHTML = text;
alert( "html="+node.innerHTML ); // Pops up message saying "html="
I tried numerous methods to fix this problem, including setting the property before and/or after adding the DOM node to the document. I also googled it which flagged up a number of related posts but these generally referred to pages that were served as XHTML (ie. pages with an .xhtml extension and MIME type application/xhtml+xml) and were not offering any solution.
The general consensus towards working around this problem is to set a timer to retry setting the innerHTML value again. (e.g. as mentioned in this page) Such a solution is highly inconvenient and is generally inapplicable, however, because it screws up your programming model - you essentially need to turn the simple operation of setting innerHTML into a callback-like asynchronous operation.
A Probable General Solution
element.appendChild(document.createTextNode("Baby you're a firework!"));
Then, as I'm using jQuery Mobile, I can patch it into jQuery's html function:
Wednesday, May 18, 2011
You can get the code here, and below is a quick screencast.
Edit: Turns out it was unnecessary. I kept searching for Mobile Firebug before I came to the conclusion that I need to write one myself. But someone from the PhoneGap user group told me there's actually a Mobile Web Inspector out there. It's good enough for me for now, although I have to say I don't really like the Java server code in there... looks overly complicated to me.
Monday, May 9, 2011
Martin-Kous-MacBook-Pro:~ martinkou$ sudo easy_install -U Cython
Searching for Cython
Best match: Cython 0.14.1
Running Cython-0.14.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-5NQPRk/Cython-0.14.1/egg-dist-tmp-gOLK4r
warning: manifest_maker: MANIFEST.in, line 19: 'recursive-include' expects <dir> %lt;pattern1%gt; %lt;pattern2%gt; ...
/usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed
Installed assemblers are:
/usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64
/usr/bin/../libexec/gcc/darwin/i386/as for architecture i386
/tmp/easy_install-5NQPRk/Cython-0.14.1/Cython/Plex/Scanners.c:6694: fatal error: error writing to -: Broken pipe
lipo: can't open input file: /var/tmp//ccAtkL0J.out (No such file or directory)
error: Setup script exited with error: command 'gcc-4.2' failed with exit status 1
This error pops up for a lot of build scripts besides Python distutil's - Perl modules like ImageMagik are also failing to build on OSX after Xcode4 is installed. The reason for this is Apple has removed the PPC assembler for OSX platforms in Xcode4, yet a lot of existing build scripts for OSX are building universal binaries.
There're quite a few solutions on the Internet, but the cleanest solution I've found so far, is the second solution (as of today) in this Stack Overflow page.
So it turns out Apple has still left a PPC assembler in the iOS platform files for Xcode 4. All you need to do is to make 2 sym links at the appropriate place and it'll work.
$ sudo ln -s /Developer/Platforms/iPhoneOS.platform/Developer/usr/libexec/gcc/darwin/ppc /Developer/usr/libexec/gcc/darwin
$ sudo ln -s /Developer/Platforms/iPhoneOS.platform/Developer/usr/libexec/gcc/darwin/ppc /usr/libexec/gcc/darwin
互聯網站 AirBnB在 SXSW後雖然確立了宣傳手法和自我特色，但仍然是小型搞作，每次只是供應幾十間住房，談不上賺錢。而談不上賺錢，對於創業者是大問題，因為幾個創辦人在三藩市的日常開支，絕不便宜。
AirBnB三人靈機一觸，由早餐想到穀物片，由穀物片想到穀物片背後的品牌和廣告，再由穀物片品牌想到總統大選。於是美國的電視上就出現了：「 Obama O’s： Hope in every bowl！」及「 Cap’n McCain’s： Put a maverick in your morning！」
利用網上宣傳的便利，美國人對總統大選的瘋狂，和前兩次搞 AirBnB網站所認識的傳統媒體， AirBnB推出了以奧巴馬和麥凱恩為主題的早餐穀物片。每款限量推出 500盒，每盒售價 40美元。 1000盒穀物片送 200盒給記者。其餘 800盒讓 AirBnB的名字除在美國迅速走紅之外，還幫 AirBnB賺了三萬美元。
「我對你們的計劃信心不大。」 08年尾， AirBnB三人去了考 Y-Combinator。考 Y-Combinator的原因並不是因為 AirBnB成功，而是因為他們沒錢。賣穀物片的 3萬元只是還了部份信用卡欠款。在全國電視一輪風光過後， AirBnB網站依然沒人流，沒收入，很快便被人忘記。折騰了一年，結果碌爆信用卡，晚餐只好吃麥凱恩和奧巴馬。考 Y-Combinator，就是想問 Paul Graham拿兩萬元給自己吃飯。
「但我欣賞你們那種大無畏的精神。不怕死，能在最壞的情況下看出並抓緊機會，比任何計劃書更好。」 Paul Graham由 AirBnB網站的統計數字觀察到，除大型會議以外， AirBnB的主要客源來自紐約。於是， Paul Graham給 AirBnB的頭號建議是，不要老是坐在矽谷，去紐約見你們的客戶。打電話給他們，敲他們的門，和他們一起拍照、開派對，用盡所有方法，令客戶記得你。
(刊於 2011-02-26 香港蘋果日報)
Thursday, February 17, 2011
AirBnB的故事在 07年 10月開始，那是 IDSA（ Industrial Designers Society of America）在三藩市的大型展覽會前夕。創辦人之一的 Brian Chesky辭掉原有的工作，搬到三藩市和朋友 Joe Gebbia一起居住。目的是到矽谷附近的三藩市，嚐嚐創業的滋味。
當時三藩市的酒店都因為展覽會全部爆滿。 Joe和 Brian覺得，這個時候出租家裏多餘的空間給短期住客，應該有市場。二人建一個簡單的網站，讓附近的人可以出租家裏的空間給參加展覽會的短期旅客，當中包括 Joe和 Brian家裏的三張氣墊床。網站一做好，兩人就以電郵主動聯絡一批與工業設計有關的博客。
不消幾天，一些和展覽會主題有關的大網站，就出現了有關 AirBnB的報道。 Joe和 Brian也如此幫自己找到 3個短期租客，賺了約 1000美元的租金。更重要是兩人體驗到短期租務市場的潛力，確立了日後透過社交媒體宣傳方向。展覽會結束， Nathan Blecharczyk加入 AirBnB。他們用同樣的方法，攻打 08年 SXSW藝術節。宣傳同樣由博客網站入手，先找小博客，報道過就找多點人看的博客，報道一輪就聯絡更多人看的網站。如是者， AirBnB早期的宣傳方法便成形。
那時的 AirBnB還有一個問題，要找住房，其實早就有 Craigslist和 Couch Surfing。 AirBnB當時的報道雖多，但只得幾十間住房，要擴張就得找個令自己與別不同的地方。
Joe和 Brian在這個時候，用自己的網站租了一間房來住，目的是了解客戶用 AirBnB租房整個過程的感受。矽谷稱這種做法做吃自己的狗糧。
這樣吃一吃狗糧， AirBnB團隊學到要避免在租住過程中談到租金， AirBnB就得要求租客先行付款。如此房東可以更放心放租，租客亦不會再受租金困擾， AirBnB亦可以從中收取佣金。比別人優勝的用戶體驗，成為 AirBnB的特色。
(刊於 2011-01-29 香港蘋果日報)
Tuesday, February 15, 2011
The testing method I used was to run a TCP client and a server in the same process and let them ping-pong each other, and measure the rate of ping-pong requests. What this means is basically the absolute best reaction times for gevent and libevent in terms of TCP socket events. As a control experiment I also did a gevent queue ping-pong between two greenlets.
The source code of the two simple tests can be found below:
libevent test case:
gevent test case:
And here are the results -
Macbook Pro 2007 (2.2GHz Core 2 Duo/4GB RAM/OSX 10.6.6)
Martin-Kous-MacBook-Pro:testgswitch martinkou$ ./benchmark
25441.49 roundtrips per second
Martin-Kous-MacBook-Pro:testgswitch martinkou$ ./benchmark.py
test_tcp_socket: 8722.40 roundtrips per second
test_queue: 265788.63 roundtrips per second
Xeon Server (Xeon X3450/8GB RAM/Ubuntu Server 10.04)
105852.88 roundtrips per second
test_tcp_socket: 14124.71 roundtrips per second
test_queue: 430830.17 roundtrips per second
So, despite gevent's awesome performance among all Python concurrent I/O libraries - it still has plenty of space for improvement. This is actually a pretty good news for the developers of gevent and similar concurrent I/O libraries in Python. Because it means the more awesome things are still to come.
Tuesday, January 18, 2011
facebook新年市值衝上 500億美元，一間 80後開的公司市值高過港股火車頭，這是現在不少人對科技的理解：錢。但之前講過，在矽谷創業及投資的人非常多。在朱克伯格以外，還有很多故事。
問以上問題的是 Y-Combinator創辦人 Paul Graham。他說你的腦筋有問題，很明顯你要見心理醫生。
作為創業者，願景荒謬是常見的事。甚麼 Google或 Apple，現在馬後炮說他們聰明很容易，但在他們開始的時候，其實都很荒唐。七、八十年代走去史丹福 Homebrew Computer Club的人，不少是長髮長鬍子，滿身臭味的嬉皮士，有些甚至會吸大麻。那時候的微型電腦，只是一個盒，沒螢幕沒鍵盤，只有一排排的開關和燈。即使是那班 嬉皮士，也不知道有甚麼用。
話說回頭，想做那個網站的人，很明顯是儍的。剛巧， Paul Graham都是儍的，所以， AirBnB這個團隊考進了 Y-Combinator。
AirBnB並不是考入 Y-Combinator才開始。 08年的 airbnb.com，是 AirBnB團隊第三次搞同類型的網站，前兩次都是失敗的。正常人的反應是放棄，一般人根本不會試三次，或失敗一次就放棄。
三個人為了試第三次，碌爆信用卡。但今時今日， AirBnB在美國的影響力足以令傳統酒店業震驚。單是紐約， AirBnB可找到的單人住房已超過 4000間，而且不限於低檔住房。 AirBnB的住房現已遍佈 168個國家， 8000多個城市。 2010年 11月， AirBnB向 Sequoia Capital和 Greylock Partners籌了 720萬美元的 Series A投資資金。
這類故事在矽谷多次發生。第一、二次人們會覺得神奇。幾十年來屢屢發生，是因為人們會隨着時間學習，發展出系統。在每一個 AirBnB疑似儍呆子變老闆神話的背後，其實都有一個實在而合理的故事。下回再講 AirBnB的故事。
(刊於 2011-01-15 香港蘋果日報)
Saturday, January 1, 2011
So, here's my story so far, in short...
- Joined a high-ranking open source project, FCKeditor as Core Developer (2007)
- Made it to computer mags (2008)
- Made it to newspapers (2009)
- Made it to TV (2009)
- Went to Beijing and caught a glimpse of the central government's ambition with the high tech industry in China (2009)
- Traveled to Silicon Valley and lived for six months (2010)
- Made it into two Silicon Valley incubators almost simultaneously and accepted one of the offers - SSE Labs (2010)
- Produced two prototype demos within the six months in SV, one of them was made public (SayCheeze), demo-ed to VCs. (2010)
- The other idea is more promising... so I'm working on that ;)
- Hooked up to an editor from Apple Daily HK which allows me to write column articles in weekends. (2010)
It's quite surprising what a determined enough person can do in 3 years. But still, I'm very far from Mark Zuckerberg...
... and that kinda puts in perspective, how big this world is. Mark is only 1 year older than me. A Forbes ranking is not too significant in my eyes - plenty of people can get insanely rich by being lucky - but a hand that guides the path of technology and thus, civilization, is significant. And I mean all these in a concrete, pragmatic sense, in case you think I'm just another dreamer.
Space is big. You just won't believe how vastly, hugely, mind- bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to space. - Douglas Adams
譬 如一間拿過天使投資，開始做出產品雛形，有合理盈利模式，但之前團隊沒有甚麼往績的新公司，剛開始找風險投資時，通常會拿到一個 Seed Round或者 Series A。這時公司的估值通常是數百萬至一千萬美元， Seed Round投入的資金，通常是幾十萬至一百萬美元，而 Series A則由一百至幾百萬美元。
Seed Round和 Series A的作用是，給公司一至兩年生存時間，在產品開發上和市場上建立相當實力。 Series A之後的科技公司會有很多不同去路，需要再擴張，可繼續籌 Series B、 C等，這些時候投入的資本，通常會過千萬甚至逾億美元。
另一個可能性是，被較大的企業如 Google、 Apple、 Facebook或 Amazon等收購，價錢視乎情況。近年大型企業收購小型科技公司例子越來越多，金額通常在數百萬至數千萬美元。
藉收購、擴市場網羅人才大公司收購小型科技公司的理由，除了是擴展市場（如近月美國 Groupon Inc.收購香港 uBuyiBuy），另一理由是招攬人才。用幾百萬美元來招聘人才，聽來荒謬，但是一個具執行能力、能夠兼顧市場、營運及技術的團隊得來不易，和普通人員招聘不可相提並論。
(刊於 2010-12-25 香港蘋果日報)
一般人很多時會以為科技公司就是簡簡單單搞一個網站，寫一個 iPhone app，多人用就可以賺錢。但是縱觀整個科技工業，由八十年代的 Apple、 Microsoft，到九十年代的 Yahoo!、 Amazon，甚至二千年的 Google、 Facebook或 Salesforce等等，無論是企業家還是投資者，其實都已經累積了好幾十年的人才、人脈和經驗。現在的矽谷，無論是創業還是融資，都已經趨向標準化和 系統化。
譬 如 3個剛畢業的大學生自行創業，獲得 Y-Combinator之類的「孵化器」垂青，他們可以預料 Y-Combinator會給他們數萬美元的資本，換大約 6%至 7%的股份，也即是數十萬美元的估值。其他的孵化器可能會用多一點錢，換多一點股份，又或者相反。但是不同的孵化器，對團隊的要求、給予的幫助及對新公司 的估值，都有不少類似的地方。
剛開始創業的人除了找孵化器之外，還可以找天使投資者。這些人通常是之前創業成功或者是大型的企業，如 Salesforce或 Cisco創始團隊中的成員。
天使投資者和孵化器相似之處，是他們都比較重視團隊的質素，概念和商業模式反而不是太重要。天使投資者通常會給數萬至十數萬美元的資本，換 2%至 10%的股份，也就是說 100百萬至幾百萬美元的估值。相對孵化器來說，找天使投資者的難度和要求會比較高。
(刊於 2010-12-18 香港蘋果日報)