Free and Flexible Development Tools

Often, when I'm talking about the shabbiness of Microsoft software I hear something to the effect that the development tools are great, especially Visual Studio. Really? Is that true or does it just appear that way since the environment is closed?

I used to agree with this assertion, but what did I know? Like many, if not most, Microsoft developers, I hadn't worked with any other development tools. As a matter of fact, there was a time when I couldn't even imagine any room for improvement. Something that I've come to notice is there is a Microsoft induced reality distortion field that is far stronger than that of any other vendor or government and it makes you lose your imagination.

Visual Studio actually does very little for you that you can't find elsewhere, and maybe even less.

One of the worst things about Visual Studio is it's opacity. It's very difficult to know what's really going on under the hood. References and components are visible only through tiny dialog boxes which in turn rely upon the registry to actually find and use the components and libraries that appear there. It's possible to see the references if you view the relevant project files as text, but the references contain elaborate numbers that map to the registry. This is a trip down the rabbit hole if ever there was one.

When using Visual Studio you have no choice about the compiler you can use or the debugger. If you want to use a real version control system you have to get third party software working on all the seats.

Intellisense is cool, but it exists elsewhere. Debugging, immediate windows and break points are standard fare in other integrated development environments.

But why belabor the point on a feature by feature comparison?

Let's take a look at a simple and useful task and see if it can be done in Visual Studio. Imagine you've got a directory filled with VB files (*.cls, *.bas, *.frm, etc.) and you want to find all the instances were a particular procedure is called, let's call it 'frmOSIssue.LoadData. How would you produce a list of all the places where this appears in the project?

The answer is you can't. Visual Studio has no facility to present you with this kind of time-saving information and Windows carries with it no tools that can accomplish this task either. In Unix/Linux/OS X, it would be accomplished with a single, simple command using software that is installed by default in every system.

grep -i frmOSIssue.loaddata

This might seem arcane, but it's no more arcane than any other weird stuff software developers have to know to do their jobs. This command calls grep, which searches text files. The -i argument tells grep to be case insensitive and the last argument is the string you want to find. Once you know what grep does and are familiar with a few of the options you're set to go. You can certainly benefit from knowing regular expressions, but you don't need to know very much to start using the command to good effect. Here's what the result looks like after it searched over 600 files in less than one second:

OSIssue.frm:    strProcedureName = "frmOSIssue.LoadData"

TLIssue.frm:    strProcedureName = "frmOSIssue.LoadData"

WizOS_P01.frm:    strProcedureName = "frmOSIssue.LoadData"

WizOS_P07.frm:    strProcedureName = "frmOSIssue.LoadData"

Similarly, I can do this in a file filled with SQL scripts to show dependencies. Basically the same command does a lot of heavy lifting that SQL Server Enterprise Manager and Visual Studio can't do.

But, but, but.... what about Visio? How can you diagram this? Many believe there is no replacement for Visio, especially since Visio's file format is so strictly proprietary that no other software can produce or read VSD files, as far as I know. First, there are other tools that provide UML and ERD. These are not as hard to come by as some would have you believe. Second, these alternatives often produce better output. Third, they're free and use simple text files, so you'll always have them and you can do other things with them too.

One diagramming package that I love, Graphviz, draws diagrams from a text file. For some really impressive examples of what this can do, check out http://graphviz.org or Google for grpahiz examples to see the possibilities. These files are simple text, so they are easy to make, easy to share, small and they will work with any software package that can read this standard format, .dot. Because they are text, it is also simple to use diff to compare one version of your diagram to another. Try doing that with Visio. Also, these packages can produce a number of graphics formats, most usefully PDF. Producing PDF is typically a built-in, globally available capability in operating systems other than Windows.

This is a little Perl script that reads the files in a directory and finds all the interrelations among them and writes a .dot file that can be processed into a graph.

#!/usr/bin/perl -w

use File::Basename;

# -- Take the list of file names from the directory

# -- Loop through the list of function names and search all files in the directory for it

# -- Put the result (filename:functionname) into an array

opendir(DIR, ".");

@files = grep(/\.sql$/,readdir(DIR));

closedir(DIR);

foreach $file (@files)

{

        $funcname = basename($file, ".sql");

        @relatedfunctions = qx{grep -l -w -m 1 $funcname *.sql};

        foreach (@relatedfunctions)

        {

        # The mapping shows that the first function listed is called by the second function listed

                $_ = basename($_, ".sql");

                if ($_ ne "$funcname\n")

                {

                chomp($_);

                print "$funcname" . " -> "  . "$_" . ";\n";

                }

        }

}

It then writes a simple file, a piece of which is here:

digraph FunctionMap {

subgraph cluster_Functions {

label=Functions;

}

AGGR_PPU_PooledAmount -> AGGR_PPU_AssignedStatus;

AGGR_PPU_PooledAmount -> AGGR_PPU_Name;

AGGR_PPU_PooledAmount -> Bond_OutStanding_AGGR_PPU;

}

After putting a little business at the top of the file it can be opened by a piece of software that reads .dot files and creates high quality diagrams. I typically use Graphviz for this, but there are other software packages that do this too.

Here is the nice, clean data-driven diagram. 

One thing I like about this is that there is no need to rearrange the boxes, it's all done for me. My staff sometimes spent the better part of a day trying to arrange the elements of a large schema by hand so that it was intelligable - no more. There are variety of styles to choose from and I'm not finished yet plumbing the depths of this format.

This is actually super efficient, light weight, platform independent and produces really great output. Is it Visio? No, it's not. But I don't miss Visio either; it's too cumbersome and expensive. The benefits of these tools outweigh the capabilities of Microsoft's IDEs. I still have to use Microsoft's IDEs sometimes when supporting some old applications because the compilers aren't readily available otherwise, but I chafe under their limitations now and use alternatives when ever possible. Also, when I need a capability I'm free to put one together myself or find it on the Internet. So far I've written scripts to convert TSQL into PL/pgsql, convert .dot diagrams into code stubs and to globally modify comments and to outline training documents.

Another example of how some basic tools can beat Visual Studio comes when you have to compare entire directories. If your directory consists of all the project files from Visual Studio or a bunch of SQL scripts and you want to compare it to another directory Visual Studio isn't going to be much help, but diff will be. The followig commands will tell you what's different between two directories of files:

diff -rq dirA dirB > dir_diffs.txt

or from within one of the directories you want to compare you can use 'find'.

find . -type f -not -exec cmp {} /path/to/dir2/{} ";" -print

Software developers should start programming new languages in their rawest form so they know what's happening, but they should move to an IDE soon after that so they can be more productive. The question is, what is the IDE? Is it something Microsoft provides for you - take it or leave it - or is it something you put together yourself based on your needs, tastes and budget? I prefer the later. Having control of my environment makes me free and independent. When I'm free and independent, I'm most productive.

What Happens in Vegas...

Some of my past speaking engagements came up in conversation recently and out of curiosity I paid a visit to the web site of the publication that held the conferences, Advisor Media. I gathered from the site that they have largely switched their focus away from technology and no longer hold the kind of conferences I used to speak at and attend.

While I was reviewing the site a blog by the publisher, John L. Hawkins, caught my eye. It was titled, "How I Escaped From Windows Vista" and it expressed his frustration and disappointment with Vista and recounted his fall back to XP. It was a good read. The few comments expressed quite similar complaints, so I just had to chime in myself.

My comment can be found at the blog, but I thought I would fill in a couple of gaps. The Advisor Media's tech conferences I was involved with were largely Microsoft affairs, though once I spoke at an interoperation conference in Maui's Ritz Carleton (very, very nice). While I certainly did a lot of Microsoft stuff at the conferences, I also specialized in Oracle and did a day long project management post conference session.

As I mention in my comment to John's post, I last spoke for Advisor Media in Vegas in '05. I had a talk on Oracle, gSOAP and I did my post conference project management session. I showed up with my PowerBook and did my thing. When I needed Windows, I ran it under Virtual PC. I was a misfit for sure and some of my old conference acquaintances did not know what to make of me. How had I gone so wrong? I had some explaining to do, but I think I was seen more as a misguided curiosity than anything else. Besides, the talks were good and showing gSOAP's performance advantage over .NET SOAP services was a hit.

Trouble loomed at the end of the conference when the speakers and representatives from Microsoft formed an expert panel to take questions from the attendees. There were a lot of questions about Share Point, which was pretty new then, and Longhorn, which was delayed again. I really couldn't speak to either of these things. I had never used Share Point, still haven't and I think it would have been pointless, and bad form, to bring up Unix/Linux/OS X in response to a Windows operating system question. After all, these folks are probably pretty much stuck with the OS whether they like it or not. So I had very little to say.

Then an attendee asked the Microsoft representatives about the state of Access' Jet engine which hadn't been updated in years. Now, regardless of what you think about Access, or what I used to think about it, Access was very important to a lot of people. Rightly or wrongly, it put food on the table for a lot of software folks, myself included, and they were genuinely concerned by Microsoft's neglect of this product.

I don't recall exactly what the Microsoft reps said about updates, but I remember that the answer ticked me off because it wasn't really an answer and I felt I had to address this attendee's question about getting his hands on an adequate database. Panelists share a microphone and pass it around, so I called down for it. I got a couple of looks from the other panelists who thought the crazy uncle was about to recite his medical history over Thanksgiving dinner, maybe it was my imagination.

I said quite simply that there was no reason to wait for Microsoft to take care of a product that is obviously so important to their work, but was essentially a commodity, there were options. I mentioned Postgresql and MySQL and how they were more than adequate and were always being updated. Anyone who had worked with databases would have no problem working with these and they would be glad they did. I think I briefly related some of my own experiences. Essentially, I said that Microsoft had turned its back on developers and they should return the favor if they weren't happy.

The other panelists were not happy about what I had to say and they were quick to get the microphone away from me if I recall correctly. Since I was up on the stage I couldn't get a real read on the audience's reaction, so my thanks go to John for letting me know that it wasn't well received. I didn't say this to piss people off, I just felt that they deserved a better answer than the one given to them by Microsoft.

It was remarkable to me that things had come to this. This was around the time when a sweaty, chair chucking Steve Balmer stormed across a stage shouting "Developers! Developers! Developers!" At the time we were supposed to believe that he was telling developers how important they were to Microsoft, but I think he was aiming those chairs at them to shut them up for asking for better tools. To Microsoft, developers exist to sell licenses.

When they take away your cheap, though effective, tools like Access or start charging for Visual Studio, when they leave the operating system bereft of compilers, languages, interpreters, debuggers, build tools, version control or file management tools so that even the simplest things require the purchase of new software or enormous effort on your part, and when the hardware requirements continue to outpace the useful life of a computer then it should be plain for all to see that they don't give a shit about developers unless you're selling licenses and feeding beast.

If Microsoft really cared about its developers in the trenches, you'd be able to start programming real software as soon as the OS booted for the first time. A technology ecosystem that gives good tools to developers is the technology ecosystem that will be advanced by those developers. New and imaginative software will be created there. Think about that for a moment. When you've got a fresh install of any version of Windows, what can you do with it? Nothing. There's no shell, Perl, Python, C++ compiler, web server, web languages or application server. There's no way to simply manage directories full of *.bas or *.sql files. You can't diagram anything, you can't really edit anything but the crudest documents in the crudest way.

The dirth of tools for developers is probably the main reason that advances in computing have never occurred on Windows and likely never will.

On the other hand, the open source environment is built by and for developers. That's why all the things I mentioned above are available to you when you boot into a clean install, you may only need to issue a simple one line command to obtain them. The availability of these tools is the reason why open source operating systems such as Linux, FreeBSD and OS X have eclipsed Windows in performance, features and security for several years. All the interesting work is being done there and has been for a very long time. There have been no significant developments coming out of Windows. Multi-user systems? Unix. Protected memory? Unix. True multiprocessing? Unix. Hardware abstraction? Virtualization? Unix. Application servers? Unix. World Wide Web? NeXT, Web Servers? Unix. Security? Started on and is still strongest in Unix. Film production? OS X. User interface advances? OS X and KDE. File management systems? Solaris and Linux. Garbage collection and managed code? Solaris. I could go on, but you get the idea.

I'd like to think that I spoke truth to power, but I didn't really. I certainly spoke the truth, but Microsoft really isn't powerful anymore. No one is really afraid of them. And those to whom software is important don't really think of Microsoft very often. Between the competition from Googel, Apple, Linux, sanctions from the EU, dismissal from standards bodies and their own incompetence they really don't matter very much at all. When I read that John has left everything behind except for Widows itself then I know something has changed - for the better.

Microsoft is like that drunken sorority sister who's had too much to drink and doesn't realize the party is over. She spent the evening dissing everyone, but now she's shouting for affection and looks as though she's about to pee on the rug. The only difference is that the girl is dressed and Microsoft has no clothes.