230 The Matrix: Code Screen

Description

Morpheus?

This week’s quiz comes from BR:

Emulate a code screen from The Matrix (also known as Falling Code or Digital Rain, etc.).

Have fun!

Summary

This summary was written by Jean Lazarou.

As nobody submitted any solution, I wrote one.

Before trying to imagine how to implement the digital rain involving some random generation of falling characters, I tried to find if the code screen in the movie had some meaning. It appears that it represents something like viewing the source code of the matrix complexity. Therefore I decided not to use any kind of random generation. Instead, I wrote a source code viewer displaying the content as falling character sequences.

The main part should obviously have been the GUI drawing, leading to a solution very specific to some drawing layer. Such a solution would not be of a big interest from the Ruby perspective. I decided to write a solution having different levels of interest, from Ruby parsing, easy extension loading, to using JRuby, the JVM interpreter, and an overview of the Java2D layer, the drawing layer.

Levels of interest

The summary contains several parts, choose the one you want and skip the other ones:

  1. overview of the final result
  2. parse Ruby scripts with any standard-compatible distribution
  3. easily load extensions to your application, here the extensions add parsers for other languages
  4. extend the classes of the APIs we use, without changing the source code
  5. introduce JRuby
  6. use Swiby, Swiby tries to make the use of Swing (the Java GUI layer) a bit simpler to use with Ruby
  7. introduce Java2D, the standard 2D drawing APIs included in the JVM

I tried to make the code readable, I made a lot of refactoring so that reading the implementation should not be too complex.

The final result

Let’s first see a snapshot of the running application.

The application expects a Ruby script as argument and displays the code in green on a black background. It uses three different kind of green colors, one for literals, one for keywords and a third one for other type of elements. It displays keywords in bold and a brighter color for the first letter. The code scrolls automatically from top to bottom.

The scrolling display could be improved very much, but I tried to keep the solution shorter and not too complex.

Don’t expect to read the scrolling code, it is totally unfriendly for humans…

Parsing Ruby code

Parsing Ruby code in Ruby is possible with several libraries you can find. I didn’t need a real parser I only needed a tokenizer. A tokenizer, or a lexer, reads source code and breaks it into tokens. Usually the tokenizer produces tokens of different types depending on the language elements.

After searching how I could tokenize Ruby code for a while I thought that rdoc, the tool that comes with the Ruby distributions generating the documentation, does parse Ruby code. I dug in the code and found how to use the tokenizer.

Say we want to tokenize a file named hello.rb and print out the tokens.

01 require 'rdoc/rdoc'
02 
03 content = File.open('hello.rb', "r") {|f| f.read}
04 
05 tokenizer = RubyLex.new(content)
06 
07 token = tokenizer.token
08 
09 while token
10   
11   puts token
12   
13   token = tokenizer.token
14   
15 end
16

First line requires rdoc. The lexer uses a string containing the code, therefore line 3 loads the file in memory. Line 5 creates the lexer hidden in rdoc. Next we start a loop through all the tokens, when the lexer reaches the end it returns a nil value.

Usually, we won’t use such a hidden code. Using functionality that is not public is not a very good idea…

Load extensions

I decided to add support for some kind of extensions or plugins to open the application to tokenizers for other languages than Ruby.

The tokenizer used to parse the code is unknown to the application. The application loads all the scripts found in the tokenizers subdirectory. Each script should extend the CodeTokenizer class, by doing so it is automatically registered as a tokenizer. Each tokenizer class answers to the question: do you parse some given script? The application instantiates the first tokenizer that accepts the script.

The code for CodeTokenizer looks like:

01 class CodeTokenizer
02   
03   @@tokenizers = []
04   
05   def self.inherited child_class
06     @@tokenizers << child_class
07   end
08 
09   def self.create file
10     
11     tokenizer = nil
12     
13     @@tokenizers.each do|tokenizer_class|
14       
15       if tokenizer_class.parses?(file)
16         tokenizer = tokenizer_class.new(file)
17         break
18       end
19       
20     end
21     
22     tokenizer
23     
24   end
25   
26   def self.parses? file
27     false
28   end
29   
30   ...
31   
32 end

The most important part of the code is the class level inherited method (line 5). Ruby calls inherited of a class each time another class extends it. Here, the code adds all the child classes to a class level variable (line 6).

The client code calls the class level method named create (line 9), a factory method. create loops through all registered child classes asking if they do parse the file (line 15) and creates an instance of the first one that does (line 16). The base class (CodeTokenizer) does not parse any file, the default behavior of parses? is to return false (line 27).

The CodeTokenizer class implements some generic methods, like loading a file. It also defines the public interface.

01   def initialize file
02     @file = file
03   end
04   
05   def tokenized_code
06     
07     content = load_file
08     
09     create_tokenizer(content)
10 
11     tokens = split_to_tokens
12 
13   end

The instance initialization for tokenizers requires a file as argument (line 1). The tokenized_code method is a template method used to retrieve the tokens collection.

To add support for a new language, create a new Ruby file in the tokenizers directory. The new file should define a new class deriving from CodeTokenizer and implement the parses? class level method, the create_tokenizer method and the split_to_tokens method.

Here is an extract from the Ruby tokenizer.

01 class RubyTokenizer < CodeTokenizer
02 
03   def self.parses? file
04     file =~ /\.rb$/
05   end
06   
07   def create_tokenizer content
08     @tokenizer = RubyLex.new(content)    
09   end
10 
11   def split_to_tokens
12     # the tokenizing loop...
13   end
14

A code sample using the tokenizer mechanism would be:

01 tokenizer = CodeTokenizer.create('hello.rb')
02 
03 tokens = tokenizer.tokenized_code

Extend the lexer’s classes

The part of the application that displays the scrolling code expects tokens that return their value as text, they must implement a text method. They must also provide some basic information about their type (keyword, values or anything else). They must implement the keyword? and value? methods.

As we use the tokens produced by rdoc, we don’t have those methods. Fortunately, they all have the same base class and thanks to the open-class characteristic of Ruby we can easily fulfill the contract.

01 module RubyToken
02 
03   class Token
04     
05     def keyword?
06       self.is_a?(RubyToken::TkKW)
07     end
08     
09     def value?
10       self.is_a?(RubyToken::TkVal)
11     end
12     
13   end
14   
15 end

The base class is named Token inside the RubyToken module. We don’t need to define the text method because it already exists.

We can say that we are hacking. We should not abuse of this, creating wrapping classes or our own classes for the tokens would be a recommended approach…

What is JRuby

I guess that presenting JRuby has nothing extraordinary today. You can find information about JRuby and how to install it on their site.

In short, JRuby is a Ruby runtime running on top of the Java Virtual Machine (JVM).

The code presented above can run with the MRI. I used JRuby because I wanted to use the GUI layer coming with the JVM.

The way JRuby integrates with the Java platform is very nice. Here is a very short list of interesting things:

Create the GUI

The GUI part could be pure Java/Swing and would be quite a lot to present it in this summary. I preferred to use Swiby to make it easier. Swiby is an ongoing project and the site is not reflecting the current version, refer to the list of demos.

Creating and opening the application’s window is simple:

01 renderer = ScrollingCodeRenderer.new(code_as_tokens_by_line)
02 
03 frame {
04   
05   width 900
06   height 600
07   
08   title script
09   
10   @display = draw_panel(:resize_always => true) { |graphics|
11 
12     graphics.background Color::BLACK
13     graphics.clear
14     
15     graphics.rotate(-Math::PI / 2)
16     graphics.antialias = true
17     
18     renderer.paint graphics
19     
20   }
21   
22   visible true
23   
24   every(frequency_millis) {
25     renderer.forward
26     @display.repaint
27   }
28   
29 }

Line 1 creates a renderer for the collection of tokens. The renderer is responsible for drawing the falling code.

The application’s window definition starts at line 3. A frame is a normal window. Lines 5 and 6 set the initial window size and line 8 sets the window’s title, it just shows the name of the loaded script.

Line 10 adds a draw panel which is the component on which the rendering happens, it needs a block to perform the painting. Line 22 makes the window visible.

Line 24 initiate a timer that is going to execute the given block every frequency_millis milliseconds. The code makes the renderer move forward to the next painting step and then triggers the painting.

The draw panel executes the given block each time it needs to paint its content passing the graphics object. The graphics object is a wrapper added by Swiby around the real Java Graphics object. A Graphics object gives access to a lot of drawing methods that, roughly, fill the image buffer sent to the screen. The Graphics object prevents from painting outside the visible area. Line 13 clears the panel in black as set at line 12. Line 15 rotates the canvas, the code draws the lines horizontally but they appear as falling.

Draw the falling code

Java offers several ways to render text, because of our needs we must revert to lower level APIs.

We define two Font objects, both are monospaced, one in normal style and one in bold for the keyword tokens. We also define three levels of green color (one for keywords, one for literals and the last one for anything else). We define them in a module so that we can include it in several classes.

01 module RendererStyles
02 
03   GREEN = Color.new(123, 232, 76)
04   DARK_GREEN = GREEN.darker
05   LIGHT_GREEN = Color.new(123, 255, 96)
06 
07   FONT = Graphics.create_font('courier', Font::PLAIN, 14)
08   BOLD_FONT = Graphics.create_font('courier', Font::BOLD, 14)
09   
10 end
11 
12 class ScrollingCodeRenderer
13   
14   include RendererStyles
15   
16   ...
17   
18 end

A Font object can transform a string to a sequence of glyphs. A glyph is a character expressed as a Shape, a shape is a suite of elementary drawing directives (like lines). Using shapes, we can apply all the available transformation to drawing, like rotate and translate.

We transform the tokens to glyph shapes but, because a glyph does not contain any color information, we need to store the shapes and the colors in a class we name TextCompositeShape.

01     glyph_sequence = font.create_glyph_vector FONT_RENDER_CONTEXT, text
02     
03     TextCompositeShape.new glyph_sequence, color, width, text

The create_glyph_vector method belongs to the Java2D APIs. It returns a GlyphVector object , the sequence of glyphs, which is not a vector we can use as normal Java arrays. It offers methods to access each glyph, change their position and apply transformations. We are going to apply a rotation transformation on each glyph to make each character appear horizontally.

We change the above TextCompositeShape creation.

01     glyph_sequence = font.create_glyph_vector FONT_RENDER_CONTEXT, text
02     
03     width = rotate_glyphs(glyph_sequence)
04     
05     TextCompositeShape.new glyph_sequence, color, width, text

Here is the implementation of the rotate_glyphs method. It returns the width of the whole sequence because we need it when we later paint it.

01   def rotate_glyphs glyph_sequence
02     
03     x = 0.0
04     spacing = 4.0
05     
06     n = glyph_sequence.num_glyphs
07     
08     n.times do |i|
09       
10       transform = AffineTransform.new
11       transform.rotate Math::PI / 2
12       
13       gm = glyph_sequence.get_glyph_metrics(i)
14       pos = glyph_sequence.get_glyph_position(i)
15 
16       pos.set_location x, pos.y - gm.bounds2D.width / 2
17       
18       x += gm.bounds2D.height + spacing
19       
20       glyph_sequence.set_glyph_position i, pos
21       glyph_sequence.set_glyph_transform i, transform
22       
23     end
24     
25     width = x
26     
27   end

Line 6 and 8 show that we cannot use usual Ruby iteration because we are playing with Java objects. Line 10 and 11 show how we define the rotation transformation we want to apply to each glyph.

Line 18 may seems odd because we add the glyph height to a x coordinate, usually we add widths. We are rotating the characters and placing them one beside the other, the distance between characters ends being the initial height.

We saw how we convert one token to a sequence of drawing shapes. We apply the same conversion to the whole set of tokens. We create an array of sequences by line of source code.

I use the term convert instead of transform so as not to get confused with the affine transform (Java2D concept).

We only need to make the token to shape conversion once and because it does not belong to the painting process, I created a class responsible for the conversion: TokenToGraphicShapeConverter.

01     @lines_as_shapes = TokenToGraphicShapeConverter.new(lines_as_tokens).convert

When it comes to paint, the code uses the lines_as_shapes variable and is rather simple.

01   def paint graphics
02 
03     y = 0
04 
05     @lines_as_shapes.each do |line_as_shapes|
06       
07       y += @height
08       x = @animation_position
09       
10       line_as_shapes.each do |composite_shape|
11       
12         graphics.color composite_shape.color
13         graphics.draw_glyphed_text composite_shape.glyph_sequence, x, y
14 
15         x += composite_shape.width
16         
17       end
18 
19     end
20 
21   end

We iterate through all the lines (line 5) and for each line we iterate through all the code segments having different styles (line 10). We change the coordinates starting at 0 for y and at the current animation position, producing the scrolling effect, for x (remember we are rotating the canvas, x becomes y).

Conclusion

As I said, the result is far from good, a lot of improvements are necessary:

I covered a lot of things and they give a basic idea in case you ever run into one of them.

To run the application, after installing JRuby and downloading Swiby, move to the directory of the application and execute from the command line:

jruby -I<swiby dir>/core/lib/ code_screen.rb hello.rb

You can download the code here.

Monday, March 22, 2010