Tokens.md (1160B)
1 +++ 2 title = 'Tokens' 3 +++ 4 # Tokens 5 ## What is it? 6 a token is a sequence of characters that represents a unit 7 8 - float literals: 3.14, 0.274e2, 42 9 - operators: +,-,*,/,% 10 - parentheses: () 11 12 tokenising is the first step of a compiler run 13 14 ## Representing/storing tokens 15 storing as chars/strings is not handy, need to still be extracted 16 17 predefined types from C++, but nothing that suits a token. so define own with classes. 18 19 ## Implementing tokens 20 token consists of kind and value. value is only used for numbers. 21 22 ```cpp 23 class Token { 24 public: 25 char kind; 26 double value; 27 } 28 29 Token t,t2,t3; // declare tokens 30 t.kind = ‘+’; // token 1 is a + 31 t2.kind=‘8’; // indicating number 32 t2.value=3.14; // value 33 t3=t; // copy initialisation 34 ``` 35 36 Now, `(1.5+4)*11` can be shown as: 37 38 <table> 39 <tr> 40 <td>'('</td> 41 <td>'8'</td> 42 <td>'+'</td> 43 <td>'8'</td> 44 <td>')'</td> 45 <td>'*'</td> 46 <td>'8'</td> 47 </tr> 48 <tr><td></td> 49 <td>1.5</td><td></td> 50 <td>4</td><td></td><td></td> 51 <td>11</td></tr> 52 </table> 53 54 ## Using tokens 55 ```cpp 56 Token get_token(); 57 vector<Token> toks; 58 59 int main() { 60 while (cin) { 61 Token t=get_token(); 62 tok.push_back(t); 63 } 64 } 65 ```