Artagnon's Presence

This article is intended to be a continuation of Modern Thoughts on Open Source. Although this article is totally independent, I’ve used the idea of “open source as an infrastructure” heavily.

Everyone understands how Microsoft makes money, but very few understand why they’re getting a wonderful operating system like Ubuntu free of cost. I’ve tried to explain it here.

The problem with opensource software is simple. The “vegetable vendor” model that most closed-source software uses fails with opensource. Take the most popular and simplest example: RedHat. It essentially creates two operating systems from the same codebase: RHEL and Fedora. They sell support for RHEL to corporates, while giving away Fedora to the community, and sponsoring its development. Make no mistake: RHEL is opensource as well, but it has no community like Fedora does. RedHat decides what to go into RHEL, and they build it themselves. Anyone else is free to build it for themselves ofcourse, but they don’t support those builds [1].

Why bother with Fedora at all? Simple. The first thing a RedHat service engineer does when he gets a complaint from a customer: attempt to replicate the bug on Fedora. If the bug’s fixed in Fedora, fixing it in RHEL is as simple as backporting the changes. If it isn’t, they’ll file a bug in Fedora. Depending on the situation, either they’ll get the bug fixed free-of-cost by the community, or will have to fix it themselves. In other words, they’re just getting code from the community free of cost by maintaining Fedora. Then why not give Fedora to the customers? Because bleeding-edge development takes place on it, and stuff often tends to break and get fixed at a later date [2]: RedHat cannot control what gets committed to Fedora, but they will make sure that only mature features get into RHEL. This service model not only applies to complete distributions, but to individual software packages as well- PostgreSQL, MySQL, Django are just a few examples.

So finally, who writes Fedora? The Fedora Board consists of nine members: five of whom are RedHat employees and four of whom are community members [3]. A lot of mainstream opensource software has a similar distribution of paid/ community developers.

Another famous model is the dual-licensing model followed by MySQL AB (now acquired by Sun). They give away MySQL community edition for free, but their enterprise edition is proprietary [4]. The advantage of having these two editions is similar to the one RedHat gains from having RHEL and Fedora. A similar approach is followed by OpenOffice.org: Their proprietary product that earns them revenue is called StarOffice. Advertising is yet another way: Google pays Mozilla for example.

So, the final question remains: apart from the corporates, who is this “community” that writes the code that end-users get free of cost? Most free software projects started out as “pet projects” with little monetary incentive. The Linux kernel was written by a student as a personal project, and several parts of the GNU operating system was written by Stallman in an attempt to create a free operating system- both were individuals who weren’t concentrating on revenue at that time. Things have changed a LOT since then. While there are still many tiny pet projects that are developed by a few people for little/ no monetary incentive, majority of the contribution to mainstream software such as Linux is from people with monetary incentive. The corporates these people work for benefit either directly or indirectly from that opensource project. Corporates that provide support for these opensource projects, that build their products on top of these opensource products, and those that simply use these products in their everyday work. Simple example: Google finds a bug in Chrome, and finds out that the underlying reason is WebKit. After fixing WebKit, getting that fix merged into upstream will benefit them, because they’re not the only people developing it. Same reason Google will contribute to SSH: They’re using it, and they’re not the only ones developing it.

Even non-monetary incentive doesn’t mean no incentive. Contributing to opensource is a great way to secure one’s future. All the work is out there in the public for everyone to see, and you don’t have to explain the work you did behind the closed doors of a large corporation in flowery language. Code talks. Everything else is CV-noise.

The whole model is sustainable in my opinion. A large corporate uses several bits and pieces of opensource software to create a product. They create an opensource project to drive it, and appoint some developers to head the project. I’ve already discussed how they’ll make money to drive it. Students and other prospective employees start contributing to the project for no monetary incentive, with the intention of rising higher in the project’s hierarchy. They’re essentially making themselves employable by the company- once the company recognizes some brilliant developers, it starts paying them and giving them specific tasks. An additional bonus kicks in when other companies also become interested in the project and start paying more developers to develop it. There’s always some ratio of developers working for monetary benefit to those working their way up.

Few people understand open source. Even fewer use this knowledge to their advantage. On one hand, there are free software fundamentalists with their utopian views, and on the other hand we have closed corporates who don’t possibly see how open source could work. Both miss the point. I’ve attempted to explain it using four questions to guide the article. Admittedly, a lot of what I’ve said here was explained to me by Kiran Jonnalagadda (aka jace, jackerhack).

1. Why do RedHat, IBM, and Novell contribute so much to the Linux kernel? Is it because they care about the free software philosophy and are being charitable? To quote the Linux Foundation August 2009 issue, “Over 70% of all kernel development is demonstrably done by developers who are being paid for their work” and “None of these companies are supporting Linux development as an act of charity; in each case, these companies find that improving the kernel helps them to be more competitive in their markets”
2. Why does Google pump money into a competing against Firefox with an open source alternative, Chrome? To quote an excerpt from their December 2009 blog post, “Today’s open source goes far beyond the “patent pooling” of the early auto manufacturers, and has led to the development of the sophisticated software components — Linux, Apache, SSH, and others — upon which Google is built”
3. Why are several of Microsoft’s products losing market share? Why does Microsoft feel threatened by Linux? To quote Ballmer, “Linux is a cancer that attaches itself in an intellectual property sense to everything it touches”. Is it because the general public is convinced by the GNU philosophy to “share with thy neighbor”, or is it simply because of competition?
4. And finally, a slightly off-topic question to reinforce my point. Why did Apple finally make the move to x86 even after their innumerable claims that PPC is better? Is it because PPC is inherently inferior to Intel’s x86, and nothing could have been done to save it?

All this points to one thing: Open source is an infrastructure. Just like the roads in the city that automobile manufacturers are interested in, software companies want access to existing infrastructure to build their own products upon and make money, instead of having to re-invent the wheel everytime. It’s a win-win situation for all companies. Paying Microsoft to get Windows to support their products can be an expensive affair. So making an operating system a commodity is in everyone’s interests (well, except Microsoft’s). The web browser is another example of a commodity today. Everyone’s interested in a good, conformant, open platform on which to develop their web applications on.

There are commodities in other sectors too. Apple moved to x86 because it was the standard commodity to use. Too many companies pumped money into it resulting in its development, and mass manufacturing brought down costs. Apple understood this- one company pumping money into PPC cannot match the rest of the world pumping money into x86.

The day a product comes out, every other company tries to clone it and make the product a commodity. In the small time window between the product being niche and the product becoming a commodity, there is money to be made. Take Netscape’s example: Once they felt the pressure of the competing web browsers, they realized that keeping their browser closed and exclusively pumping money into it was pointless. So they opened it out, and started the Mozilla project. In other words, open sourcing a piece of software typically means that that piece of software can no longer make money as a traditional product anymore. Same reason Facebook open-sourced many of its components: It has everything to gain, nothing to lose. It’s already too big on the social networking front to worry about competition there.

Largely, there are two ways to make money off this infrastructure: sell the cars that use these roads most sensibly, or follow a “pure service” model. The problem with the pure service model is that only RedHat and a few other companies have been successful at it. The argument “I created it, so I can service it best” doesn’t hold anymore- too many people have technical expertise in open source software. So unless your service is the cheapest and the best, it’ll be killed off by other service providers.

Products don’t have to start off being closed and then open up when they stop making money. Companies are also interested in starting off free software projects. Google Chrome for example. They wanted to make this web browser a commodity, a commodity created by them. They’ll have a foot at the door when Chrome gains larger market share.

I thought the “Haskell Internals” talk was probably too technical. I expected a lot of people to have some prior exposure to functional programming. Otherwise the Haskell code dump would have shocked them. @swaroopch told me that my “An Insight into CPython Compiler Design” (it’s called unladen-swallow.pdf here) was good, mainly because I made it clear what I’m going to be talking about in the title. @NOLFXceptMe told me that the talk was super-informal, like I was explaining to someone across the coffee table- exactly what I intended! It has a LOT of C code, but that didn’t scare people one bit. I could tell from the feedback session that many of them actually read the code, unlike the Haskell code from my previous talk. Anyway, lesson learnt. Here are the slides. You can get the sources and compile it for yourself with notes, or just see the BiBTeX references here. Oh, and the comments section in the blog exists for a reason- I want feedback! I can’t improve without it.

haskell-internals.pdf (504 KB)

unladen-swallow.pdf (644 KB)

Unladen Swallow is “an optimization branch of CPython, intended to be fully compatible and significantly faster”. Interesting. In order to understand Unladen Swallow, we first needed to understand the CPython compiler. It’s  written in C and fairly easy to understand, very unlike the self-hosting complex beasts Steel Bank Common Lisp and Glasgow Haskell Compiler. The design of the compiler is explained fully in PEP 339, and I’m just including a small warm-up discussion.

Start off at the top level: the evaluator (Python/cevel.c). It executes Python bytecode. From the Python program you write to the Python bytecode that is produced, there are several stages with optimizations at every stage. It goes like this:


1. Using the SPARK parser generator, generate the Parse Tree from the Python code  (Parser/pgen.c)
2. Convert the Parse Tree into an Abstract Syntax Tree (Python/ast.c)
3. Transform the AST into a Control Flow Graph. Several compiler optimizations like removing unreachable code are implemented here (Python/compile.c)
4. Post-order DFS the CFG and flatten it to emit the final Python bytecode. The bytecode is then subjected to various peephole optimizations (Python/compile.c)
5. Instead of having to go through this this long procedure everytime we want to run the same program, the Python bytecode produced is stored in a .pyc file.

Enough warming up. The Unladen Swallow developers want to optimize CPython very smoothly, preferably without breaking any existing code. Their Project Plan is laid out on the Wiki page; I’ll mainly discuss 2009Q2.

If they work at any stage higher than the Python bytecode, they’ll have to re-implement the fantastic peephole optimizations written by the community over the years. The plan is simple: instead of executing the high-level bytecode in the eval loop (which can be painfully slow), compile atleast some part of it into a lower representation that can be executed faster. Why not compile all of it to x86 assembly? Firstly, we’d lose the interactive shell, a very major part of Python. Secondly, we’d lose out on Python’s high portability, that arises because it’s not tied down to any single architecture’s assembly language (I even run Python on my phone using the PyS60 package). Thirdly, it’s terribly difficult to do, makes Python uglier and unmaintainable, and it’s utterly pointless- we’re trading off too much for execution speed.

Just-in-time (JIT) compilation is the solution. Compile the code only when needed. Then again, don’t compile everything; compile when the new execution time is less than the old execution time by an extent enough to justify the compile time. Otherwise, the startup delay will become unbearable and the interactive interpreter won’t really be interactive anymore. The perfect low-level platform independent representation to JIT to is LLVM (Low-level virtual machine). So, the Unladen Swallow developers first wrote a compiler from Python bytecode to LLVM IR from scratch (Python/llvm_compile.cc). The LLVM IR (Intermediate Representation) looks a lot like assembly, except that it’s architecture-independent,  and can be subject to a nice set of optimizations. Then they modified the eval loop to execute LLVM IR, while keeping the existing Python bytecode evaluator (file renamed: Python/eval.cc).

Q3 and Q4 aren’t updated on the Wiki page, but it should be easy enough to find out the work in progress from the IRC channel, mailing list, and repository commits.

This course attempts to teach programming methodology from an engineering viewpoint, as well give an insight into theoretical computer science.

Guidelines:

1. Teach programming, and not the programming language. Discuss abstract ideas in class, and ask students to write implementations in a language of their choice. Avoid building language-specific patterns (for example, the double-for loop for generating prime numbers in C got stuck in my head). Discuss various different implementations AFTER the students have tried it themselves.
2. Discussions should span several programming paradigms. Initially, cast problem statements to strongly indicate a paradigm. For example “Go over numbers from 1 to 10 and print those that are even” versus “Define an even number and print even numbers <= 10″. Later, let students figure it out themselves.
3. Develop a hacker mindset- give students incomplete solutions written by other students and ask them to complete it.
4. For routine assignments, prepare a huge problem set of ascending difficulty to randomly pick from, along the lines of Project Euler. Problems should cover algorithms, pure math, computational geometry, and dynamic programming.
5. Conduct collaborative programming sessions where students learn to implement from each other. No single instructor will be able to provide such a vast variety of implementations.
6. Prepare a reading list for software engineering, including articles on editing (introduction to specialized editors such as Vim and Emacs), debugging, versioning, testing, and profiling. It’s important to make students write large programs collaboratively in steps, so that they will be able to appreciate the importance of these tools.
7. Prepare another reading list for theoretical computer science. Also give students sneak peeks into the latest work in every field. The purpose of this exercise is to get students excited, so that they can work independently to eventually discover their research interests.
8. Finally, it all boils down to grading. If you’re going to ask students to evaluate things like (i++) + (++i) in the exam, it’s clear that you’re expecting students to know programming language intricacies, and not programming methodology.

lecture1.pdf (130 KB)

curve-fitting.pdf (168 KB)

I delivered a talk on curve fitting today in the physics department seminar room. This is the presentation I used. The objective of the talk was to familiarize everyone with the chi-square distribution and chi-square fitting without focusing too much on the implementation details. Since the complete audience didn’t have background knowledge of probability theory, I had to briefly touch up some concepts like Cumulative Distribution Function and Probability Density Function.

Overall, I think the talk came out alright, although it was apparent that some of my fundamentals were holed. It’s a miracle it even came out, considering that I was delivering it on a paracetamol-suppressed headache and fever.

Finally, LaTeX + Beamer is too awesome!

Edits: The zeroth central moment is one, not zero.

Download now or preview on posterous

draft-1sep.pdf (122 KB)

While going through some unused parts of my hard disk, I found
something pretty fascinating. I wrote this out in 2007 to explain
pointers in C to someone. Hopefully, someone somewhere will find it
useful :)

Posted via email from Ramkumar’s posterous

No time limit, No word limit. Write as much as you can. This is the test paper I prepared to select students for the tech division of Entrepreneurship Cell, IIT Kharagpur.

Page 1/4

Page 2/4

Page 3/4

Page 4/4