latest changes: 

comments do not start inside symbolic identifiers i.e.

+//~ is a single token "+//~" but a//~ is "a"

note "::" and "=" are defined internally and cannot be rebound (because of possible ambiguities)

number literals, faster more efficient lexical parsing

fixed bug where input like "1[+2" was read as "1+2"
and similar bug for "fun f x (y = x + 1"

base unit kg for M
unit gram = 1e-3 kg;


To do list:

Add support for datatypes, including type constructors in AtPat (more than just OpAtPat)

defntok

inverses?

move functions from ValueFun to rich.miffs.functionlibrary ?

value browser (!)

units, dimensions

matrices

>>FINISH replace arith.compare...  with Value.compareTo(Object)

sort out return values. e.g. "val a=1; val b = 1/0" doesnt print the value of a, but does set it

ln / log

make vals return a list of string<->val with special formatting.

powers of functions

overloading?

datatypes?

-----------------

identifiers:
  aplhanumeric: any sequence of letters, digits, primes (') and underscores (_) starting with a letter or prime
  symbolic: any non-empty sequence of 
     ! % & $ # + / : < = > ? @ \ ~ ` ^ | *
  minus:
     -
     
reserved words are not identifiers e.g. => is not an identifier, but =>> is

Reserved words:
   _ ( ) [ ] , . ; if let fn

Semi-reserved words: (these words act as their respective syntactic keywords whenever possible, otherwise they are free to be used as identifiers)
   then else | => = :: andalso orelse 
   

/* the BNF grammar used is :
   <input> ::=  ( <defn> | <exp> ) [ ";" [ <input> ] ]

   <defn> ::= "val" <at_pat> "=" <exp> 
            | "fun" <varname>_1 <pattern>_1 "=" <exp>_1 
              [ "|" <varname>_1 <pattern>_2 "=" <exp>_2 ]...
            | "infix"  [<int>] <id> [<id>]...
            | "infixr" [<int>] <id> [<id>]...
            | "nonfix" [<int>] <id> [<id>]...
            | "baseunit" <id> [<id>]...
                
   <pattern> ::= <at_pat> [ <pattern> ]

   <at_pat> ::= <at_pat> "::" <at_pat>
              | <id> 
              | "_"
              | "(" <at_pat> [ "," <at_pat> ]... ")" 
              | "[" <at_pat> [ "," <at_pat> ]... "]" 
              | <at_pat_const> 

   <at_pat_const> ::= "()" | "[]" | "true" | "false" | [ "-" ] <number>

   <exp> ::= <exp2>
           | <exp2> "andalso" <exp>
           | <exp2> "orelse" <exp>
           
   <exp2> ::= "if" <exp> "then" <exp> "else" <exp>
           | "fn" <pattern> "=>" <exp> [ "|" <pattern> "=>" <exp> ]...
           | "let" ( [ <defn> ] [ ";" ] )... "in" <exp> [ ";" <exp> ]... "end"
           | <applyexp>
           // | <exp2> : ty //TODO

   //For some sufficiently large integer n, we define a family 
   //of BNF expressions, indexed from 0 to n inclusive:                   
   <applyexp> ::= <apply[0]>
   
   //for 0 <= i < n
   <apply[i]> ::= <apply[i+1]> [ <val-infix-level-i> <apply[i+1]> ]...
   
   <apply[n]> ::= <val> [ <val> ]...

   <val> ::= <tuple> | <list> | <id> | <const>

   <const> ::= "()" | "[]" | "true" | "false" | <number> | <string>

   <tuple> ::= "(" <exp> ["," <exp>]... ")"
   <list> ::= "[" <exp> ["," <exp>]... "]"
   
   
      
end of BNF

REGEXPS:
let \a = [a-zA-Z']
let \b = [a-zA-Z'_0-9]
let \x = [!%&$#+/:<=>?@~`^|*\-\\]
let \d = [0-9]

then an identifier matches (\a\b*)|(\x+)
and a number matches (\d+(\.\d*)?|\.\d+)((e|E)(+|-)?\d+)?
*/
