Merge lp://staging/~nahiljain/to-drizzle/ver-1.0 into lp://staging/to-drizzle

Hi!

OK, good work so far, Nahil! Here's some stuff to work on. I'll place the merge proposal back in Work In Progress. When you correct the things below, simply switch the merge proposal back to Needs Code Review status.

1)

You need to tell BZR to ignore certain files that PLY generates. To do so, you must remove these files from BZR and then use the bzr ignore command, like so:

$> bzr remove --force helpers/parser.out helpers/parsetab.py
$> bzr ignore parser.out parsetab.py

2)

Let's change from using commands.getstatusoutput() to direct calls to the Lexer in our test cases for the lexer.py. I'd like you to start thinking more about making the stuff in lexer.py into distinct functions that may be called from other files. In other words, don't have anything in lexer.py print to stdout. Instead, have functions return sequences of things that can be iterated over.

For example, instead of this code in lexer.py:

117 +# Test it out
118 +#data = '''
119 +#CREATE DATABASE abc;
120 +#CREATE SCHEMA `neh`;
121 +#create TABLE first;
122 +#'''
123 +if __name__ == '__main__':
124 + data=sys.argv[1]
125 +# Give the lexer some input
126 + lexer.input(data)
127 +
128 +# Tokenize
129 + while True:
130 + tok = lexer.token()
131 + if not tok: break # No more input
132 + print tok

it would be better to have the following:

def tokenize(subject):
  """Given a string subject, tokenizes the string into LexToken objects and returns them"""
  lexer.input(subject)
  results= []
  while True:
    t= lex.token()
    if not t:
      break
    results.append(t)
  return results

Then, in your test cases, you can replace this code:

491 +import unittest
492 +import os
493 +import commands
494 +class TestSequenceFunctions(unittest.TestCase):
495 + def lexer_debug_output(self, sql):
496 + (res, output)= commands.getstatusoutput("python ../movetodrizzle/helpers/lexer.py '''%s'''" % sql)
497 + return output
498 + def test_create_schema_1(self):
499 + sql = "CREATE SCHEMA test;"
500 + expected_results='''LexToken(CREATE,'CREATE',1,0)
501 +LexToken(SCHEMA,'SCHEMA',1,7)
502 +LexToken(IDENTIFIER,'test',1,14)
503 +LexToken(SEMICOLON,';',1,18)'''
504 + results= self.lexer_debug_output(sql)
505 + self.assertEquals(results,expected_results)

with the following:

import unittest
import os
from helpers.lexer import tokenize
from ply import LexToken

class TestCreateSchema(unittest.TestCase):
  def test_create_schema_1(self):
    sql= "CREATE SCHEMA test;"
    expected_results= [
      LexToken(CREATE,'CREATE',1,0)
      LexToken(SCHEMA,'SCHEMA',1,7)
      LexToken(IDENTIFIER,'test',1,14)
      LexToken(SEMICOLON,';',1,18)
    ]
    results= tokenize(sql)
    self.assertEquals(results, expected_results)

That way, you are calling the lexer.py's tokenize function directly and not having to go through the shell just to test these things. Let's reserve using commands.getstatusoutput() to testing other things related to the overall program execution. For things like this, it's better to call the library functions themselves.

3)

I'd like you to start thinking more about consistency and modularity. If you see yourself repeating chunks of code, move a chu...