C# Guru – An Interview With Eric Lippert

Eric Lippert’s name is synonymous with C#. Having been Principal Developer at Microsoft on the C# compiler team and a member of the C# language design team he now works on C# analysis at Coverity.

If you know C# then the name Eric Lippert will be synonymous with clear explanations of difficult ideas and insights into the way languages work and are put together.

Here we host an overall summary of the highlights of the interview ranging over topics as diverse as the future of C#, asynchronous v parallel, Visual Basic and more (the link to the full interview on i-programmer can be found at the end of this page), so read on because you will surely find something to interest you about C#, languages in general or just where things are heading.

NV : So Eric, after so many years at Microsoft you began a new career at Coverity. Was the ‘context switch’ easy?

EL : Yes and no. Some parts of it were very easy and some took some getting used to.

For example, re-learning how to use Unix-based development tools, which I had not touched since the early 1990s, took me a while. Git is very different than Team Foundation Studio. And so on. But some things were quite straightforward.

Coverity’s attitude towards static analysis is very similar to the attitude that the C# compiler team has about compiler warnings, for instance. Though of course the conditions that Coverity is checking for are by their nature much more complicated than the heuristics that the C# compiler uses for warnings.

Switching from taking a bus to downtown every day instead of taking a bus to Redmond every day was probably the easiest part!

NV: I guess that from now on you’ll be working on the field of static analysis. What exactly does static analysis do?

EL: Static analysis is a very broad field in both industry and academia. So let me first start very wide, and then narrow that down to what we do at Coverity.

Static analysis is analysis of programs based solely on their source code or, if the source code is not available, their compiled binary form. That is in contrast with dynamic analysis, which analyses program behavior by watching the program run. So a profiler would be an example of dynamic analysis; it looks at the program as it is running and discovers facts about its performance, say.

Any analysis you perform just by looking at the source code is static analysis. So for example, compiler errors are static analysis; the error was determined by looking at the source code.

So now let’s get a bit more focused. There are lots of reasons to perform static analysis, but the one we are focused on is the discovery of program defects. That is still very broad. Consider a defect such as “this public method violates the Microsoft naming guidelines”. That’s certainly a defect. You might not consider that a particularly interesting or important defect, but it’s a defect.

Coverity is interested in discovering a very particular class of defect.
That is, defects that would result in a bug that could realistically affect a user of the software. We’re looking for genuine “I’m-glad-we-found-that-before-we-shipped-and-a-customer-lost-all-their-work” sort of bugs. Something like a badly named method that the customer is never going to notice.

NV: Do Code contracts play a role, and will the introduction of Roslyn affect the field of static analysis?

EL: Let me split that up into two questions. First, code contracts.
So as you surely know, code contracts are annotations that you can optionally put into your C# source code that allow you to express the pre-condition and post-condition invariants about your code. So then the question is, how do these contracts affect the static analysis that Coverity does? We have some support for understanding code contracts, but we could do better and one of my goals is to do some research on this for future versions.

One of the hard things about static analysis is the number of possible program states and the number of possible code paths through the program is extremely large, which can make analysis very time consuming. So one of the things we do is try to eliminate false paths — that is, code paths that we believe are logically impossible, and therefore do not have to be checked for defects. We can use code contracts to help us prune false paths.

A simple example would be if a contract says that a precondition of the method is that the first argument is a non-null string, and that argument is passed to another method, and the second method checks the argument to see if it is null. We can know that on that path – that is, via the first method – the path where the null check says “yes it is null” is a false path. We can then prune that false path and not consider it further. This has two main effects. The first is, as I said before, we get a significant performance gain by pruning away as many false paths as possible. Second, a false positive is when the tool reports a defect but does so incorrectly. Eliminating false paths greatly decreases the number of false positives. So we do some fairly basic consumption of information from code contracts, but we could likely do even more.

Now to address your second question, about Roslyn. Let me first answer the question very broadly. Throughout the industry, will Roslyn affect static analysis of C#? Absolutely yes, that is its reason for existing.

When I was at Microsoft I saw so many people write their own little C# parsers or IDEs or little mini compilers or whatever, for their own purposes. That’s very difficult, it’s time-consuming, it’s expensive, and it’s almost impossible to do right. Roslyn changes all that, by giving everyone a library of analysis tools for C# and VB which is correct, very fast, and designed specifically to make tool builder’s lives better.
I am very excited that it is almost done! I worked on it for many years and can’t wait to get my hands on the release version.

More specifically, will Roslyn affect static analysis at Coverity? We very much hope so. We work closely with my former colleagues on the Roslyn team. The exact implementation details of the Coverity C# static analyzer are of course not super-interesting to customers, so long as it works. And the exact date Roslyn will be available is not announced.

So any speculation as to when there will be a Coverity static analyzer that uses Roslyn as its front end is just that — speculative. Suffice to say that we’re actively looking into the possibility.

NV: What other possibilities does Roslyn give rise to? Extending the language, macros/mutable grammars, Javascript like Eval, REPL?

EL: Some of those more than others.
Let me start by taking a step back and reiterating what Roslyn is, and is not. Roslyn is a class library usable from C#, VB or other managed languages.Its purpose is to enable analysis of C# and VB code. The plan is for future versions of the C# and VB compilers and IDEs in Visual Studio to themselves use Roslyn.

So typical tasks you could perform with Roslyn would be things like:

  • “Find all usages of a particular method in this source code”
  • “Take this source code and give me the lexical and grammatical analysis”
  • “Tell me all the places this variable is written to inside this block”

Let me quickly say what it is not. It is not a mechanism for customers to themselves extend the C# or VB languages; it is a mechanism for analyzing the existing languages. Roslyn will make it easier for Microsoft to extend the C# and VB languages, because its architecture has been designed with that in mind. But it was not designed as an extensibility service for the language itself.

You mentioned a REPL. That is a Read-Eval-Print Loop, which is the classic way you interface with languages like Scheme. Since the Roslyn team was going to be re-architecting the compiler anyway they put in some features that would make it easier to develop REPL-like functionality in Visual Studio. Having left the team, I don’t know what the status is of that particular feature, so I probably ought not to comment on it further.

One of the principle scenarios that Roslyn was designed for is to make it much easier for third parties to develop refactorings. You’ve probably seen in Visual Studio that there is a refactoring menu and you can do things like “extract this code to a method” and so on.
Any of those refactorings, and a lot more, could be built using Roslyn.

As for if there will be an eval-like facility for spitting fresh code at runtime, like there is in JavaScript, the answer is sort of. I worked on JavaScript in the late 1990s, including the JScript.NET langauge that never really went anywhere, so I have no small experience in building implementations of JS “eval”. It is very hard. JavaScript is a very dynamic language; you can do things like introduce new local variables in “eval” code.

There is to my knowledge no plan for that sort of very dynamic feature in C#. However, there are things you can do to solve the simpler problem of generating fresh code at runtime. The CLR of course already has Reflection Emit. At a higher level, C# 3.0 added expression trees. Expression trees allow you to build a tree representing a C# or VB expression at runtime, and then compile that expression into a little method. The IL is generated for you automatically.

If you are analysing source code with Roslyn then there is I believe a facility for asking Roslyn “suppose I inserted this source code at this point in this program — how would you analyze the new code?”

And if at runtime you started up Roslyn and said “here’s a bunch of source code, can you give me a compiled assembly?” then of course Roslyn could do that. If someone wanted to build a little expression evaluator that used Roslyn as a lightweight code generator, I think that would be possible, but I’ve never tried it.

It seems like a good experiment. Maybe I’ll try to do that.

NV:Although, the TPL and async/await were great additions to both C# and the framework, they were also cause of a lot of commotion, generating more questions than answers:

What’s the difference between Asynchrony and Parallelism?

EL: Great question. Parallelism is one technique for achieving asynchrony, but asynchrony does not necessarily imply parallelism.

An asynchronous situation is one where there is some latency between a request being made and the result being delivered, such that you can continue to process work while you are waiting. Parallelism is a technique for achieving asynchrony, by hiring workers – threads – that each do tasks synchronously but in parallel.

An analogy might help. Suppose you’re in a restaurant kitchen. Two orders come in, one for toast and one for eggs.

A synchronous workflow would be: put the bread in the toaster, wait for the toaster to pop, deliver the toast, put the eggs on the grill, wait for the eggs to cook, deliver the eggs. The worker – you – does nothing while waiting except sit there and wait.

An asynchronous but non-parallel workflow would be: put the bread in the toaster. While the toast is toasting, put the eggs on the grill. Alternate between checking the eggs, checking the toast, and checking to see if there are any new orders coming in that could also be started.

Whichever one is done first, deliver first, then wait for the other to finish, again, constantly checking to see if there are new orders.

An asynchronous parallel workflow would be: you just sit there waiting for orders. Every time an order comes in, go to the freezer where you keep your cooks, thaw one out, and assign the order to them. So you get one cook for the eggs, one cook for the toast, and while they are cooking, you keep on looking for more orders. When each cook finishes their job, you deliver the order and put the cook back in the freezer.

You’ll notice that the second mechanism is the one actually chosen by real restaurants because it combines low labour costs – cooks are expensive – with responsiveness and high throughput. The first technique has poor throughput and responsiveness, and the third technique requires paying a lot of cooks to sit around in the freezer when you really could get by with just one.

NV: If async does not start a new thread in the background how can it perform I/O bound operations and not block the UI thread?

EL: Magic!

No, not really.

Remember, fundamentally I/O operations are handled in hardware: there is some disk controller or network controller that is spinning an iron disk or varying the voltage on a wire, and that thing is running independently of the CPU.

The operating system provides an abstraction over the hardware, such as an I/O completion port. The exact details of how many threads are listening to the I/O completion port and what they do when they get a message, well, all that is complicated.

Suffice to say, you do not have to have one thread for each asynchronous I/O operation any more than you would have to hire one admin assistant for every phone call you wanted answered.

NV: What feature offered by another language do you envy the most and would like to see in C#?

EL: Ah, good question.
That’s a tricky one because there are languages that have features that I love which actually, I don’t think would work well in C#.

Take F# pattern matching for example. It’s an awesome feature. In many ways it is superior to more traditional approaches for taking different actions on the basis of the form that some data takes.But is there a good way to hammer on it so that it looks good in C#? I’m not sure that there is. It seems like it would look out of place.

So let me try to think of features that I admire in other languages but I think would work well in C#. I might not be able to narrow it down to just one.

Scala has a lot of nice features that I’d be happy to see in C#. Contravariant generic constraints, for example. In C# and Scala you can say “T, where T is Animal or more specific”. But in Scala you can also say “T, where T is Giraffe or less specific”. It doesn’t come in handy that often but there are times when I’ve wanted it and it hasn’t been there in C#.

There’s a variation of C# called C-Omega that Microsoft Research worked on. A number of features were added to it that did not ever get moved into C# proper. One of my favorites was a yield foreach construct that would automatically generate good code to eliminate the performance problem with nested iterators. F# has that feature, now that I think of it. It’s called yield! in F#, which I think is a very exciting way to write the feature!

I could go on for some time but let’s stop listing features there.

NV:What will the feature set of C# 6.0 be?

EL:I am under NDA and cannot discuss it in details, so I will only discuss what Mads Torgersen has already disclosed in public. Mads did a “Future of C#” session in December of last year. He discussed eight or nine features that the C# language design team is strongly considering for C# 6.0. If you read that list carefully — Wesner Moise has a list here

http://wesnerm.blogs.com/net_undocumented/2013/12/mads-on-c-60.html

– you’ll see that there is no “major headliner” feature.

I’ll leave you to draw your own conclusions from that list.

Incidentally, I knew Wesner slightly in the 1990s. Among his many claims to fame is he invented the pivot table. Interesting guy.

NV: Java as tortured as it might be, revitalizes itself due to Linux and the popularity of mobile devices. Does .NET’s and C#’s future depend on the successful adoption of Windows by the mobile devices ?

EL: That’s a complicated question, as are all predictions of the future.

But by narrowly parsing your question and rephrasing it into an — I hope — equivalent form, I think it can be answered. For the future of technology X to depend on the success of technology Y means “we cannot conceive of a situation in which Y fails but X succeeds”.

So, can we conceive of a situation in which the market does not strongly adopt Windows on mobile devices, but C# is adopted on mobile devices? Yes, absolutely we can conceive of such a situation.

Xamarin’s whole business model is predicated on that conception. They’ve got C# code running on Android, so C# could continue to have a future on the mobile platform even if Windows does not get a lot of adoption.

Or, suppose both Microsoft fails to make headway on Windows on mobile and Xamarin fails to make headway on C# on Android, etc. Can we conceive of a world in which C# still has a future? Sure.

Mobile is an important part of the ecosystem, but it is far from the whole thing. There are lots of ways that C# could continue to thrive even if it is not heavily adopted as a language for mobile devices.

If the question is the more general question of “is C# going to thrive?” I strongly believe that it is. It is extremely well placed: a modern programming language with top-notch tools and a great design and implementation team.

NV: Do you think that C# and the managed world as a whole, could be “threatened” by C++ 11 ?

EL: Is C# “threatened” by C++11?

Short answer: no

There’s a saying amongst programming language designers – I don’t know who said it first – that every language is a response to the perceived shortcomings of another language.

C# was absolutely a response to the shortcomings of C and C++. (C# is often assumed to be a response to Java, and in a strategic sense, it was a response to Sun. But in a language design sense it is more accurate to say that both C# and Java were responses to the shortcomings of C++.)

Designing a new language to improve upon an old one not only makes the world better for the users of the new language, it gives great ideas to the proponents of the old one. Would C++11 have had lambdas without C# showing that lambdas could work in a C-like language? Maybe. Maybe not. It’s hard to reason about counterfactuals.
But I think it is reasonable to guess that it was at least a factor.

Similarly, if there are great ideas in C++11 then those will inform the next generation of programming language designers. I think that C++ has a long future ahead of it still, and I am excited that the language is still evolving in interesting ways.

Having choice of language tools makes the whole developer ecosystem better. So I don’t see that as a threat at all. I see that as developers like me having more choice and more power tools at their disposal.

NV: What is you reply to the voices saying that C# has grown out of proportion and that we’ve reached the point that nobody except its designers can have a complete understanding of the language ?

EL: I often humorously point out that the C# specification begins with “C# is a simple language” and then goes on for 800 dense pages. It is very rare for users to write even large programs that use all the features of C# now. The language has undoubtedly grown far more complex in the last decade, and the language designers take this criticism very seriously.

The designers work very hard to ensure that new features are “in the spirit” of the language, that design principles are applied consistently, that features are as orthogonal as possible, and that the language grows slowly enough that users can keep up, but quickly enough to meet the needs of modern programmers. This is a tough balance to strike, and I think the designers have done an exceptionally good job of it.

NV: Where is programming as an industry heading at and will an increasingly smarter compiler that will make programming accessible to anyone, something very positive of course , on the other hand pose a threat to the professional’s programmer’s job stability?

EL: I can’t find the figure right now, but there is a serious shortage of good programmers in the United States right now. A huge number of jobs that require some programming ability are going unfilled. That is a direct brake upon the economy. We need to either make more skilled programmers, or making writing robust, correct, fully-featured, usable programs considerably easier. Or, preferably, both.

So no, I do not see improvements in language tools that make it easier for more and more people to become programmers as any kind of bad thing for the industry. Computers are only becoming more ubiquitous. What we think of as big data today is going to look tiny in the future, and we don’t have the tools to effectively manage what we’ve got already.

There is going to be the need for programmers at every level of skill working on all kinds of problems, some of which haven’t even been invented yet. This is why I love working on developer tools; I’ve worked on developer tools my whole professional life. I want to make this hard task easier. That’s the best way I can think of to improve the industry.

Link to the full interview on i-programmer

nikosNikos Vaggalis has a BSc in Computer Science and a MSc in Interactive Multimedia. He works as a Database Developer with Linux and Ingres, and programms in both Perl and C#. He writes articles, conducts interviews and reviews technical IT books

How I became a Python Programmer

Alex MartelliAround the turn of the millennium I was working on a C++ project. I had recently discovered the USENET newsgroups and I spent a lot of my time there. Back then, the life and soul of the Italian C++ newsgroup was a person who answered many questions with long, detailed and accurate posts: Alex Martelli.

After a while Alex started giving answers like “Yes, you could do it like that in C++, but in Python it would be so much more convenient” or “I’ll illustrate the idea using Python as pseudo-code, since it’s so easy to understand”. A seed was planted.

Learning Python

Fifth Edition

During a trip to the US I bought myself a copy of ‘Learning Python’, by Mark Lutz and David Asher – not only a very good introductory book, but also one of the few available at the time! Those were the times when Python bore a ‘1’ as major release number and the Java implementation was still called ‘JPython’.

Another encouragement came from winning the O’Reilly prize draw at an ACCU conference, awarded by my kind host, Josette. I believe I chose just about every Python book they had in their catalogue at the time.

Python in a Nutshell

Second Edition

Alex started posting less and less on it.comp.lang.c++, and started gaining fame on the comp.lang.python newsgroup, where he soon earned the ‘martellibot’ nickname, for the number and quality of his posts. And their length. When I heard that he was to write ‘Python in a Nutshell‘ I thought that it would be no smaller than a coconut shell!

Around the same time several people drifted out of the newsgroup, to resurface a little later on the newly-founded Python one. Some of them are among the founders of Python Italia, regular speakers at conferences, and contributors to popular Python projects.

Guido van RossumA few years later I met Guido at an ACCU conference. I still don’t know where I found the guts of to impose myself at his breakfast table, counting on his being too polite to send me off. I actually exploited Alex’s name as an ice-breaker, and we went on to talk of topics as diverse as build automation and exchange traded funds.

Python cookbook

Third Edition

One day, I received an email from O’Reilly asking me where should they send my complimentary copy of the second edition of the ‘Python Cookbook‘. Now, I am not above accepting unsolicited presents, but I really could not understand why they decided to send it to me. When it arrived, it was accompanied by a letter saying that it was meant to thank me for my contribution. I was even more puzzled, because I’d never even posted any recipes to the online cookbook, then hosted by ActiveState. After a while I remembered mentioning the ‘stateless proxy‘ pattern in a comment to Alex’s ‘borg‘ recipe. I bet not even Stephen King ever got paid that much by the word! It was totally undeserved, considering that I had actually heard of the ‘stateless proxy‘ pattern from one of Alex’s posts on it.comp.lang. moderated.

Years passed, versions of Python came and went, and Python 3k became reality. Now rumours about Python 4 are starting to spread… As for me I’m still a rather sporadic Python programmer, even though I find it an invaluable tool. I even managed to collect a few modules I tend to reuse and put them on SourceForge. You can find them here.

not orginalRereading what I just finished writing I realise that it sounds almost as an O’Reilly advertisement, but all these things really happened and it is a fact that O’Reilly has been publishing valuable Python resources consistently for over a decade. This is not to say that theirs are the only worthwhile Python books: years ago I enjoyed Dave Beazly’s ’Python Essential Reference‘ as a reference and lately Wesley Chun’s ‘Core Python Programming’ helped me better understand some of the recipes from Dave’s ’Python Cookbook’, third edition”, but the people who led me through my career as a Python programmer are undoubtedly Alex… and Tim.

Nicola Musati 1Nicola Musatti has been working both as developer and sysadmin for longer than he cares to admit, occasionally falling to one side or the other. He loves Python because it lets him tackle system administration tasks with the developer’s mindset. Other than Python he has programmed professionally in C, C++, C#, Java and various shells.

All about Perl 6 – interview of Jonathan Worthington (Part 3 of 3)

perl butterflyNV: The JVM has been primarily designed with statically typed languages in mind. The same goes for the CLR, and that is why the DLR (build on top of CLR) came into existence.  Have you at some point considered the DLR (maybe combined with Mono instead of the CLR) or JVM’s Dynalink, both of which admittedly have a good Meta Object protocol infrastructure, as a potential backend to Rakudo?

JW: One interesting thing to note about Perl 6 is that it’s  a gradually typed language, which means the considerations are a little different from if it was dynamically typed. In that sense, VMs which explicitly seek to do both static and dynamic typing well are especially interesting for Perl 6.

I can’t speak too well to the DLR, but I do know that Niecza, the Perl 6 on CLR implementation, went the way of not using it for a range of reasons. By contrast, the JVM’s invokedynamic instruction has been rather interesting from a Perl 6 implementation point of view.

Dynalink is certainly of interest too, in so far as it seems to provide an interesting path to enabling calling Perl 6 code from Java. I’d rather reuse an existing solution than reinvent that wheel. I need to dig into it more deeply to be really sure.

NV: Parrot is a register-based machine and MoarVM follows in its footsteps, so it looks like that this model works best. What are the merits of a register-based machine? Is it more adept to the dynamic  type of languages?

JW: It’s a bit easier to build a relatively efficient interpreter with a register-based machine. I think that was one of Parrot’s main reasons to go that way.  Even when you have a JIT compiler, being able to interpret efficiently still matters. JIT compilation takes time: you want to spend the time on hot paths, not on something you hit once in the program’s lifetime.

Having now done the code generation work for register and stack machines (since the JVM is stack based), it doesn’t feel in any sense to me that the dynamic nature of the language has much to do with it. Mostly, I found the register-based code generation a bit easier to do, because in Perl 6 most constructs can show up as an expression.
To take an example, a try block in Perl 6 works just fine as an expression. But on the JVM, the stack must be empty either side of a try/catch construct. So you have to do some work to spill the stack into locals if there’s anything already on it.

I haven’t really come out of all this thinking register machines are overwhelmingly better than stack machines, though – but they are a bit easier on the code generation, which is nice. If you plan to put your call frames on the system stack,  the stack approach seems more attractive from that angle. Once you decide you’re not doing that, you could go either way. I chose the easier code-gen, because hey, laziness is a virtue, right?

NV: Interestingly enough, although VMs are more and more embraced, increasingly locking the programmer into the managed world abstracting him away from low level languages like C, most of them are themselves written in C! How would the IT world  be affected if a virus outbreak wiped out all  C programmers in a day?

JW: Pretty badly, though anybody who managed to reverse engineer the language would probably be able to make some good money training new C programmers!

While such a virus outbreak sounds more like the plot for an awful movie than a realistic concern (though,  I would say that because  I’m a C programmer), there is some reason for concern: I’m not sure that many people are being taught C these days.

I ask a lot of the people I teach what languages they encountered while studying. Java is up there, but C I don’t hear so much. Now, from the point of view of what’ll land a job, that probably makes sense. But knowing C properly implies knowing how a bunch of low-level things work.
So, perhaps there is something to fear in this regard, even if it’s not a C programmer-eating virus!

NV: How does that sound for  a B-movie scenario: dead C programmers return as zombies and get back at the high-level programmers!

JW: How many brains will they eat before segfaulting?

NV: So what is Perl 6’s place in the programming world? What niche can it excel at?

JW: For one, I suspect there will always be text to be transformed, unstructured data to be untangled, and so forth. Perl 6 is strong at this: grammars bring structure to parsing problems and let you get from text to data structure easily, and the range of tools for slicing and dicing that data are immense.

One of the things I find most pleasant about working with Perl 6 code is how easy it tends to be to refactor and evolve. Depending how far you want to go, a quick code sketch can be grown into something more structured – perhaps with classes and roles if needed –  and for added safety you can add in type constraints. Not just boring ones, like “it’s an Int” –  thanks to subset types you can be very expressive.

I think it’s hard to predict exactly what niches a general purpose language will find itself in, however. I mean, it’s not like Perl was designed from the outset to be good for web programming, but when the need came Perl was there and had the ingredients.

Perl 6 has a lot of ingredients to offer. The language seems to sell itself rather well, even if the implementations aren’t all the way there yet. So I’m looking forward to see what people will cook up with it, as we gradually remove more and more of them adoption barriers.

NV: You’re a polyglot speaking Perl  and C#,  hacking on compilers and VMs, teaching, travelling, giving talks at conferences – you seem tireless!   Are you an android that thinks in binary and lives on a limitless power source? Where do you draw that energy from and how do you recharge it?

JW: No, I’m decidedly human with plenty of fragilities. And sometimes, I very acutely feel that. I suspect the most important thing is that I just really enjoy a lot of the things I do. Giving talks is something I really enjoy doing; it gives me energy at some level. All the same, however much you like it, work has a tendency to be, well, work. And eventually, there’s a need to recharge.

I tend to do that by taking  holiday  where I can spend a lot of time outdoors, because my work sadly keeps me indoors a lot. The Swiss Alps are an almost perfect escape, offering hours of beautiful hiking followed by a hearty meal, a nice pint, and some good rest.

NV:  Nowadays there is a lot of buzz around Perl 5 and 6 with the likes of p2, p11, Moe, Perl 7, Perlito, ports to JVM and now MoarVM. Are we witnessing a fallout, separatism, or a diversified attempt of the Empire to strike back and reclaim the throne of the programming Universe?

JW: I think we’re witnessing a community applying the “There’s More Than One Way To Do it” principle to itself. I think it’s tempting to worry about over-diversity, but I find it heartening that the community is of a nature where instead of throwing about ideas as just words, we actually tend to take those words and rectify them into concrete software. This in turn helps us better understand what going in these various directions would look like.

I don’t know where all of these projects will be in 5 years, 10, years, 15 years. But I don’t find myself worrying about it too greatly either. At the end of the day, it’s about producing good technologies that solve problems, and hopefully finding some joy in doing so. Inevitably, some things will work out well, others won’t. But there doesn’t have to be winners and losers. We can, as a community, learn much from every one of these efforts, and that knowledge will leave us richer overall.

 

nikosNikos Vaggalis holds a BSc in Computer Science and a MSc in Interactive Multimedia. Professionally, he is engaged in the development of database applications and anything related to the RDBMS. He is a SQL aficionado and codes in both Perl and C#. As a journalist, he writes articles, conducts interviews and reviews technical IT books

All about Perl 6 – interview of Jonathan Worthington (Part 2 of 3)

perl butterfly

NV: What does that say about the JVM’s project’s life expectancy? Will it run in parallel with MoarVM’s development? Does all this forecasts Parrot’s demise?

JW: The JVM is playing that difficult “second backend” role. Going from one target VM to two exposes all kinds of assumptions, such as places where things are not sufficiently well abstracted and so forth.  Such abstractions are not easy to design because you’re not only thinking about how to hide differences, but how to convey enough semantic information downwards in order to allow good code generation.

Having the second VM that  is already mature and well established is a huge help. OK, I did manage to segfault the JVM thanks to the odd invokedynamic bug I managed to hit upon. But by and large, if something is wrong, I can be pretty sure that the JVM itself will not be to blame. So, the JVM porting leads the way.

That naturally means we’ll get to a rather complete Rakudo on JVM some way ahead of getting there with MoarVM. The work on the two is going on in parallel. The JVM work has already blazed the “how to port Rakudo” trail, so really the MoarVM project complexity is much more tied up with the VM itself, not with working out how to get things ported to it.

As for Parrot: I think it’s worth remembering that Rakudo, along with the module ecosystem out there, runs on Parrot today. Of course, we aim to get there with the JVM and MoarVM in the coming months. But whatever happens with Parrot, those using and developing Rakudo will probably still be running it on Parrot for some  time to come. So that’s the short term. In the mid-term, the user base will decide what they prefer to run Rakudo on, and the answer will vary between users. We’ve no plans to go ripping out Parrot support once we run on the JVM and MoarVM. The path genuinely is multi-backend.

NV: Marketing wise, aren’t you concerned that announcing the development of yet another VM for Rakudo Perl will  reignite doubt as well as  sustain the commotion over Perl 6 and its future ?

JW: Yes, though to put it into context, the doubt around Parrot itself has not been especially helpful from a Rakudo marketing perspective either. The reason MoarVM was built in secret up to the point it was revealed, was partly a marketing consideration. Coming out saying, “Well, we have this idea to build yet another VM” would have been rightly met with a lot of scepticism. Saying, “Oh, we have this thing you can already translate and run over 80% of the NQP (a Perl 6 subset used to write Rakudo) test suite on” is a rather stronger position.

So yes, it concerned me. Deciding to work on MoarVM was not easy  for that reason and others. Frankly, I didn’t need a third full time job. Don’t get me wrong, I enjoy working on MoarVM, but it’s demanding in terms of concentration and brain cycles. Not to mention that debugging a GC is not exactly a fun evening in.Rather, I did it because, having spent a bunch of years growing an understanding of Perl 6 as a language and how to implement it, I reached the point where the costs of working on MoarVM looked like they could be outweighed by the wins from doing so. Plenty of toil remains to reach the goal, but if the Rakudo developers have shown anything, it’s endurance.

NV: In a previous interview, Moritz Lenz shared with us that Parrot’s main mistake was that  instead of focusing on satisfying Rakudo’s needs, it aimed to become a general platform for dynamic languages, something that ultimately made life hard for the Rakudo implementers. Has MoarVM learned from that, or is interoperating with other dynamic languages on the agenda?

JW: Well, there’s one dynamic language we’re very interested in interoperating with, and that’s Perl 5. For now, we’ll take the expedient path of making 5/6 interop work nicely rather than trying to generalize the whole thing into some beautifully abstract set of interop primitives. But it’s not just a case of running Perl 5 on MoarVM – though the v5 module work that’s going on may allow that to – but rather a case of integrating the existing Perl 5 interpreter.

On the question of being a general platform, though, if you look at the VMs that do successfully host multiple languages today, they’ve typically started by hosting one language really well. My interest is in making MoarVM do Perl 6 really well. If, some years down the line, that makes it an attractive target for other languages… well, that’s just fine. But I’m most certainly not thinking about the needs of other languages when taking design decisions for MoarVM.
All of this said, Perl 6 has a LOT of language features. And the NQP compiler tool-chain that we build Rakudo on is rather rich and powerful. So there’s nothing to really stop people using it to write a compiler for another language today.

Indeed, writing a compiler once that targets the JVM, MoarVM, and so forth for free could be an interesting proposition. Naturally, I’ll be happy if people pick up the tools and enjoy creating other language implementations with them. But my focus when designing and building MoarVM is quite firmly – “What do I need for Perl 6?”

NV: Integrating/embedding the existing Perl 5 interpreter into the platform means that Perl 5 will keep its blood ties to XS and won’t finally get a proper hardware independent VM ?

JW: Exactly. The goal of this bit of interop is to make CPAN modules and an organisation’s existing Perl 5 investment accessible from Perl 6. You don’t go that far before you hit an XS dependency, so if the goal is to make existing stuff accessible then XS is a non-negotiable. All that said, there is an active effort to implement Perl 5 on top of the same compiler tool-chain that Rakudo is built on top of. It’s early days yet for the work, but it is making good progress.

That bit of work is known as v5, by the way. And it wasn’t something coordinated top-down from those of us already working on Rakudo for years. It was a new contributor who said “Hey, let’s try to make this work!” The good news is that it should be fairly easy to get that effort running on the JVM and later MoarVM also.

NV: Can or will there be alternative tools that could emit bytecode for MoarVM?  For example could MoarVM be considered a possible backend to Perlito?

JW: The bytecode format is specified, and the goal is for it to be declared stable a little way down the line, so there’s no real reason why not. One thing I decided fairly early on is that MoarVM will not have is an official intermediate language or assembly language.

The code generation we do today for the NQP to MoarVM translator goes directly from a tree data structure down to bytecode, skipping the costly generating and reparsing of text. So if you want to do that from outside of MoarVM,  you’d actually need to generate bytecode. Or maybe somebody will build a little assembler.

NV: The I/O infrastructure is currently based on ARP.Why are you considering a move from ARP to libuv? Is it because of the node.js-like asynchronous execution capabilities? How will those be integrated into the platform?

JW: Yes, it’s for the asynchronous I/O. Rakudo’s lack of support for that has been a common and very legitimate complaint for a while, and I know it’s an adoption blocker for some, so clearly I’m interested to deliver something there. We’ll be able to do it on the JVM, and MoarVM should certainly handle that case too.

We could build what we need from scratch, but why? I mean, things like the object model and JIT compilation make sense to be done in a very custom way. They’re the core domain, the places that MoarVM can shine by doing what Perl 6 needs efficiently. For async IO, there’s no reason to recreate something when we simply want to do well there, not discriminate MoarVM from the market by somehow doing async IO better.

Of course, that doesn’t mean at the language level we shouldn’t have a better story.  But at the runtime level, taking something battled-hardened and integrating it seems the way to go.
As for how, we have a rather free hand there, since Rakudo doesn’t support async IO yet

The more interesting challenges will probably be in making async IO really nice to do at the Perl 6 language level, rather than the VM level stuff, which will certainly have it’s challenges but are more “hidden”.

NV: So how is the Rakudo-to-Java interoperation utilized on the JVM? Because of that relation can Perl 6 run on Android?

JW: The first of those is a currently work in progress, and I’m optimistic we may ship support for calling into Java code in the next Rakudo compiler monthly release. It will be interesting to see how that gets used; I know various people have expressed interest, so I’m mostly waiting to see what they will do with it.

As for Android, one current blocker is our use of the new invokedynamic instruction. Once we get a “don’t use indy” compilation option, or invokedynamic becomes available on Android, I suspect that with a little effort it should be possible.

Part 3 which concludes the interview will be published next week

 

nikosNikos Vaggalis holds a BSc in Computer Science and a MSc in Interactive Multimedia. Professionally, he is engaged in the development of database applications and anything related to the RDBMS. He is a SQL aficionado and codes in both Perl and C#. As a journalist, he writes articles, conducts interviews and reviews technical IT books

All about Perl 6 – interview of Jonathan Worthington (Part 1 of 3)

perl butterflyThe latest news is that Parrot Virtual Machine is no longer the only one enjoying Rakudo’s exclusivity.

Not long after Rakudo had been successfully ported to the JVM a new hosting candidate that aims to change the rules of the game, MoarVM, went public.

All that activity has raised a lot of questions regarding Parrot, Rakudo and the direction Perl 6 is heading for:

  • Is Parrot dead ? Why yet another VM and what’s up with the JVM?
  • How does Perl 6 stack up against other modern languages like C# ?
  • Is there any room in the programming world for it, after all?

For getting the definitive answers to these intriguing questions, we’ve managed to get hold of Jonathan Worthington for an exclusive interview. Jonathan is the architectural mastermind and primary driving force behind not only Rakudo’s implementation, but also all the projects revolving around it, such as NQP, the port on the JVM, and now the brand new MoarVM.

As if simultaneously engaging in these cognitively demanding activities was not short of a Herculean task, Jonathan also maintains a full time job, teaching classes in – what else? – programming.

The interview is not just confined to Perl-ish matters, and Jonathan also shares his expert views on a number of complementary topics, such as C#’s dynamic features, and whether C still has any relevance in programming.

NV: Wherever your name is mentioned, Perl 6 springs to mind. Did you really jump into Perl 6 straight away or did you already  have a background in Perl 5 ?

JW: I certainly used Perl 5 for many years before I was involved with Perl 6. I learned a lot from Perl. I came to it, like many, for web programming. My family suggested I get a summer job. I didn’t want to stack shelves, and it was the kind of time when it was easy to get work for building web stuff, so I reckoned I better figure out something you could do web programming with.

At the time, Perl was a pretty obvious choice for that. I got myself a copy of the Camel book expecting to learn a new language, and came out of it with a new appreciation for programming.
Perl was my first exposure to regexes, higher order programming, closures, and probably a dozen more things I simply didn’t know about before. This was before I went off to study Computer Science, of course. I was mostly a self-taught programmer. I even learned to indent my code “the hard way!”

I was, but a Perl 5 user, though. I discovered Perl 6 around the same time I was increasingly specializing in compilers, language semantics, type systems and so forth during my computer science degree. I got curious. Then I ended up contributing. I expected to do just a patch or two… well, I guess I’ve done a few more than that by now.

NV: Do you think that Perl is still a viable option for web programming when now it has to compete with the likes of PHP, Ruby on Rails etc. which own the biggest slice of the pie?

JW: For sure. The nature of the web and web programming has changed considerably since I hacked together my first CGI script almost 15 years ago. But Perl’s web story most certainly hasn’t stood still in that time.

There are multiple modern ways to do web programming with Perl: Catalyst, Dancer, Mojolicious. Honestly, I don’t do a lot of web programming these days; I feel like I’ve already done enough of it for a lifetime – but I do attend plenty of Perl conferences, and talks on all of these show up pretty often. They look modern, and pleasant.

So yes, Perl is a very, very viable option in my view. A cool and trendy option? Well, maybe not that. But the Perl community is blessed with a lot of good people who build a lot of good stuff. I value that.

NV: Aside from mastering Perl, you make a living out of teaching, classes in C#, amongst other subjects.  How did you come up with that combination, given that those two languages are diametrically opposed?

JW: If we were talking about C# version 1, back in 2000 or so, I think I’d have agreed with “diametrically opposed”. C# then felt like “another Java”, and while I greatly respect a lot of the things built in Java, I find it a difficult language to efficiently express myself in.

However, C# went places I really did not expect. These days, C# is a multi-paradigm programming language. Granted, it expects your major building blocks to be classes, but within that you’ve a lot of opportunities for higher order programming, declarative programming, and so forth.

Put that together with type inference and the result is a language I find fairly expressive.
Granted, I don’t find it expressive to the same level as Perl, especially Perl 6. But I think it’s fair to say that C# has – though I suspect much of the .Net community would look at me oddly for saying this – has become more Perl-y over the years.

As for teaching programming languages, I think having an understanding of what makes languages tick and how they fit together is important. Closures, lazy evaluation, types, objects – yes, the details vary (greatly in places) between C# and Perl 6. But knowing how these things are implemented means you can explain them pretty well for most languages.

So in that sense, I find my roles complement each other. It makes sure I can still explain to people the kinds of things I’m working on implementing!

NV: What about C#’s injected into-dynamic language features? Do they really make a difference?

JW: Are you thinking of things like the ‘dynamic’ keyword in C#? If so, I can only say that my experience “in the wild” is that most developers don’t use that. I’m not sure we should be too quick to say, “Oh, it’s because C# programmers refuse to consider dynamic typing”, however. Many people I teach are barely using Linq or lambda expressions, which showed up in C# a release before “dynamic”.

The “Ooh, that’s actually useful” moment people tend to have with dynamic is when I show them how, with the appropriate library, you can use dynamic typing in C# to dig into JSON documents, just like you would do in JavaScript. “Oh, we don’t have to build a load of types to deserialize this stuff into!” So, mostly it’s showing people how things are actually useful.

NV: So broadly speaking, what can experience in Perl offer a C# programmer and vice versa?

JW: “The limits of my language mean the limits of my world.” – Wittgenstein.
Now, I’m pretty sure he was talking about natural languages, but I think it goes for programming languages too. And, just like a natural language has a community and a value system around it, so do programming languages.
Perl and C# may both be multi-paradigm languages, but they do feel rather different to write. I found myself approaching problems in different ways in each of them. But I’m quite sure working in both has influenced the way I write each of them.

I think in the Perl to C# direction, you bring an understanding that not everything needs to be expressed as methods on a class, fitting into some type hierarchy. It’s OK for things to be “just a subroutine”, or “just a function”, or “just a quick thing you do with a regex” – because that’s exactly what some problems need.

Going the other way, I’d say you perhaps come with a little more appreciation of where adding type annotations into programs can help. I’m excited about what we’re doing in Perl 6 in this regard with gradual typing. And, if you get into C#’s Linq stuff really well, you start to see more operations as list processing, which Perl is rather good at. Perl 6 especially so.

And that’s a win, because chains of list operations are easy to read – the next thing’s input is the last thing’s output – as well as nice and functionalish.

NV: So let’s go back to the beginning and talk about Rakudo, JVM, and MoarVM. Was the move of porting Rakudo to the JVM planned, or was it driven by Parrot’s deficiencies?

JW: Having Rakudo also running on the JVM has been of interest to us for years. Python is on the JVM. Ruby is on the JVM. Where is Perl? What if Perl fits your problem, but you are only allowed to deploy things on the JVM at your organization? The modern reality is that talking about “run everywhere” is no longer just about operating systems and CPUs. Some of those “everywheres” are virtual machines now.

So there’s a lot more to it than, “Parrot has deficiencies”. Even if Parrot had worked out incredibly efficient, getting Perl 6 onto other runtimes would still have been important. The JVM should give us a speed boost for a bunch of workloads. The numbers so far are encouraging. Startup time, on the other hand, is pretty awful. I doubt we can ever get that really low on the JVM.

But perhaps the most important thing for the Perl 6 project overall is that the JVM has very well tested, battle-hardened threads support. We really, really need to nail down those aspects of Perl 6. Once the porting is over, that’s where I expect a lot of my focus will go. So, the JVM support helps fill in a bunch of the puzzle pieces.

NV: Work on the JVM port is successful and well underway, so why are we already looking at a new VM, MoarVM? Why the need to implement, for example, a GC and JIT for MoarVM when these components are readily available on the JVM? Ultimately, do we need yet another VM, or is it the case that the existing families of VMs cannot cope with Perl 6’s programming model?

JW: One of the turning points in Rakudo’s development was 6model, the name I gave to an objects implementation designed with Perl 6’s specific needs in mind. As part of the design for that, I thought a lot about optimizations – JIT included – and ensuring it would be possible to actually build a 6model implementation from “bare metal” as well as in terms of existing VM constructs.

This was initially with the idea of getting the work integrated into Parrot someday. But the more I looked at it, the more difficult that seemed. 10 years of development led by different architects at different periods of time (each of them with good ideas, I should add) led to a fair amount of baggage, as it would in any software project. With some quiet encouragement from others, I started considering what it would take to take the 6model design and put an “r” in it. So that’s where the name comes from: Moar = Metamodel On A Runtime. MoarVM is a place where we’ll be able to do “native” implementations of various Perl 6 constructs, with the idea being that a very close semantic match can lead to efficiency.

Additionally, there will be people who want to use Perl 6, but who really don’t want to use the JVM to run it. Maybe they don’t want the overhead, maybe they don’t like it, maybe they are doing one-liners or other things where you mostly care for start-up time.

One fear in all of this, of course, is that this new diversity of backends would stretch an already stretched development team. In fact, the result has been quite the opposite. People who never really cared for a Perl 6 on Parrot but really like the idea of Perl 6 on JVM, have shown up and contributed. MoarVM has drawn in a relatively disjointed set of people. So my feeling is that if there is this much diversity in potential contributors, it will exist for potential users too, since I imagine the vast majority of people contributing to Perl 6 are doing so because they want to use it for more and more of their day to day work.

Part 2 to be published next week

 

nikosNikos Vaggalis holds a BSc in Computer Science and a MSc in Interactive Multimedia. Professionally, he is engaged in the development of database applications and anything related to the RDBMS. He is a SQL aficionado and codes in both Perl and C#. As a journalist, he writes articles, conducts interviews and reviews technical IT books