The Total Growth of Open Source -
Amit Deshpande and Dirk Riehle from SAP Labs have conducted and published a research on the growth of open source software.
The data has been culled from Ohloh.net and is based on the stats and activity of around 5000 open source projects written in 30 different languages and 103 open source licenses.
Some interesting quotes from the publication:
Successful open source projects like Linux, Apache, PostgreSQL and many others are growing super-linearly. Previous research showed that linear and quadratic growth is the dominant growth pattern of open source software projects
Our work shows that the additions to open source projects, the total project size (measured in source lines of code), the number of new open source projects, and the total number of open source projects are growing at an exponential rate. The total amount of source code and the total number of projects double about every 14 months.
Open Source has taken off handsomely and continues to thrive. It is not just about philosophy any more, it is good business sense.
In case you are interested about Solr stats, see the Solr project page at Ohloh.
Update - The report is quite old but I just discovered it now :)
Erik has written about Solr’s usage in libraries on the Lucid Imagination Blog. Solr has found its way into many libraries and quite rightly so. However, one of the main things that Erik talks about in that blog post is the performance of DataImportHandler vs SolrMarc (the indexing library used by both VUFind and Blacklight).
Quoting from Erik’s email to the solrmarc-tech google group:
The difference in speeds:
SolrMarc: 22 docs / s
DIHmarc: 1,745 docs / s
W00t!
Well, I don’t know much about SolrMarc but I’ve seen DataImportHandler instances with comparable (or even better) throughput many times. There is something fishy inside SolrMarc for sure and I have a feeling that fixing it would be a low hanging fruit. However, Erik’s opinion is that DataImportHandler is a better way to index and that he will devote a portion of his time to helping the Solr using library community (thanks Erik!).
DIH has really taken off since it debuted in Solr 1.3 and it would be safe to call it the de-facto standard for indexing data into Solr. It may not be the most elegant way to index data but it is quick and it works great. With the planned features for DataImportHandler in Solr v1.5, it will continue to improve and it makes sense to base VUFind and Blacklight’s indexing infrastructure on top of it. I’m very excited to see this happening.
Exploding mobile web usage in India -
Opera published a study titled State of the Mobile Web, November 2009 which I found through TechCrunch. I can’t help but notice the tremendous growth in web usage through mobile phones in India. Page views have grown by 228.5% Y/Y and unique users have grown by 208.4% but if you look at metrics like page views per user or the amount of data transferred per user, you’ll see that they are quite small.
One of the reason is that in India we don’t have 3G (actually we do but it is limited to only one provider - BSNL) and browsing the Internet is painfully slow on GPRS connections. With 3G finally coming next year, I’m quite sure that the mobile web usage in India will just explode.
Right now the only kind of mobile applications that can work in India are SMS based. The increase in download speeds will make more people use the Internet and SMS will be less relevant, though that may take more time than a year. If 3G indeed is introduced by mid next year by prominent providers like Airtel and Vodaphone, I won’t be surprised to see Y/Y growth rate exceeding 500%.
Companies building products/services around SMS should start thinking about a mobile web strategy now.
I had been thinking about moving away from Blogger to my own domain. Finally, I decided to give in and I was fortunate enough to buy this domain. Blogger has been a simple service but I wanted to try the new kids Tumblr or Posterous. After spending some time fiddling with both of them, I decided to go with Tumblr.
It took me some time to figure out the right way to move from Blogger. I used the import script written by Jonathan Tron to import my old posts. Sadly, there is no way to import comments from blogger. The least I could do was to import them into disqus and link the same disqus account into tumblr; which actually does not help much.
The bigger issue was to migrate without leaving RSS subscribers in the lurch. The slightly lesser issue was to preserve my earlier blog’s Google page rank.
The first issue was solved easily. You can do the same with the following sequence of steps.
The second was slightly more tricky. The right way to move a website to a different domain is to use permanent redirects from the old page to the new page. However, Blogger (obviously) does not allow you to do that. Thankfully, Google recently announced support for specifying canonical links which can point to the preferred version of a URL. So, I hacked up a script to match pages of blogger’s RSS with tumblr’s RSS and generate conditional blogger template snippets which let me specify the canonical (tumblr) URL for each page on my Blogger account.
I couldn’t redirect so I had to fall back on meta-refresh to redirect anyone visiting an old page to the new page. I hate to break the back button like this but that was the only possible way in this case. This is what it looked like:
I used to think that blogging is a solved problem. After doing all this and trying out many service, I don’t believe that anymore.
[video]
AOL listed on the New York Stock Exchange on 10th December 2009. This has been in the works for a long time and I’m glad we’re finally here. Things are changing around the company and I’m happy to be a part of this change.
AOL has a new logo (and yes, it is still to be written as AOL). I loved the new brand videos, watch them on youtube - http://www.youtube.com/watch?v=YlSL7svbooY
Seed.com was also launched a few days back. It is a new spin on crowd sourcing content which, I believe, is a great idea.
We’re just getting started!
Apache Lucene Java 3.0.0 has been released. Lucene Java 3.0.0 is mostly a clean-up release without any new features. It paves the path for refactoring and adding new features without the shackles of backwards compatibility. All APIs deprecated in Lucene 2.9 have been removed and Lucene Java has officially moved to Java 5 as the minimum requirement.
See the announcement email for more details. Congratulations Lucene Devs!
The Apache Lucene project is pleased to announce the release of Apache Mahout 0.2.
Highlights include:
- Significant performance increase (and API changes) in collaborative filtering engine
- K-nearest-neighbor and SVD recommenders
- Much code cleanup, bug fixing
- Random forests, frequent pattern mining using parallel FP growth
- Latent Dirichlet Allocation
- Updates for Hadoop 0.20.x
Details on what’s included can be found in the release notes.
Downloads are available from the Apache Mirrors
From the official announcement:
Apache Solr 1.4 has been released and is now available for public download!
http://www.apache.org/dyn/
Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of
many of the world’s largest internet sites.
Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Tomcat. Solr uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it easy to use from virtually any programming language. Solr’s powerful external configuration allows it to be tailored to almost any type of application without Java coding, and it has an extensive plugin architecture when more advanced customization is required.
New Solr 1.4 features include
Note: The following material and presentation was prepared for students of the Indian Institute of Information Technology (IIIT), Allahabad. The aim was to get them excited about contributing to open source projects and in particular about Apache Lucene, Solr and Hadoop. The first talk was titled “Why you should contribute to Open Source” and was aimed at freshmen and has no technical content. The second was titled “Get involved with the Apache Software Foundation” and was given to sophomore, junior and senior students and it goes into some basic technical information on Apache Lucene, Solr and Hadoop projects. The following post comprises of some notes that I put together for the talks.
Work on what you like, when you like
Everybody wants to work on “cool” products. However, the reality is that most of you will get stuck in a job which although may pay well, it will hardly be about the things you wanted to work on. In your course, you will learn about algorithms, distributed systems, natural language processing, information retrieval, bio-informatics and other areas of computer science and its applications but in real life, the majority of the work done in software companies will have little direct application of things you will learn in your course.
Most of the times you will be using things built by others and writing glue code to build things needed by your company’s business. This is not to say that all that knowledge will go waste; it will definitely help you become a better programmer and you should learn it but there’s a fair chance that it may not be used directly in your job.
Open Source projects offer you a chance to work on something that you want rather than something that others want you to work on. It is a great opportunity to work on something that is both cool and useful as well as to associate with a well known brand and all the publicity and goodwill it brings. You are free to pick and choose between the thousands of open source projects out there. Moreover, you are free to decide on how much you want to contribute. You won’t have a boss and you won’t have the pressure of deadlines and schedules.
Development in the “real” world
Academic projects are insufficient to impart many of the skills that you’d need once you start developing software full-time. Many of these skills are “social” rather than technical in nature but are at least as important.
Most academic projects are “toy” projects. By that, I mean that their whole life cycle revolves around you. You are the designer, developer, tester and also the user. As a result, there are few key things missing in those projects.
