DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Securing Your Software Supply Chain with JFrog and Azure
Register Today

Trending

  • Event-Driven Architecture Using Serverless Technologies
  • Using Render Log Streams to Log to Papertrail
  • Revolutionizing Algorithmic Trading: The Power of Reinforcement Learning
  • Personalized Code Searches Using OpenGrok

Trending

  • Event-Driven Architecture Using Serverless Technologies
  • Using Render Log Streams to Log to Papertrail
  • Revolutionizing Algorithmic Trading: The Power of Reinforcement Learning
  • Personalized Code Searches Using OpenGrok
  1. DZone
  2. Coding
  3. Languages
  4. Creating an Editor with Syntax Highlighting Using ANTLR and Kotlin

Creating an Editor with Syntax Highlighting Using ANTLR and Kotlin

Federico Tomassetti first built a lexer and parser, and he's followed up with an editor built in Kotlin.

Federico Tomassetti user avatar by
Federico Tomassetti
·
Aug. 11, 16 · Tutorial
Like (5)
Save
Tweet
Share
9.53K Views

Join the DZone community and get the full member experience.

Join For Free

in this post we are going to see how to build a standalone editor with syntax highlighting for our language. the syntax highlighting feature will be based on the antlr lexer we have built in the first post . the code will be in kotlin, however it should be easily convertible to java. the editor will be named kanvas .

implementing syntax highlighting in the editor for your dsl

previous posts

this post is part of a series on how to create a useful language and all the supporting tools.

  1. building a lexer
  2. building a parser

code

the code is available on github . the code described in this post is associated with the tag syntax_highlighting

setup

we are going to use gradle as our build system.

buildscript {
   ext.kotlin_version = '1.0.3'

   repositories {
     mavencentral()
     maven {
        name 'jfrog oss snapshot repo'
        url  'https://oss.jfrog.org/oss-snapshot-local/'
     }
     jcenter()
   }

   dependencies {
     classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
   }
}

apply plugin: 'kotlin'
apply plugin: 'application'
apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'antlr'

repositories {
  mavenlocal()
  mavencentral()
  jcenter()
}

dependencies {
  antlr "org.antlr:antlr4:4.5.1"
  compile "org.antlr:antlr4-runtime:4.5.1"
  compile "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
  compile "org.jetbrains.kotlin:kotlin-reflect:$kotlin_version"
  compile 'com.fifesoft:rsyntaxtextarea:2.5.8'
  compile 'me.tomassetti:antlr-plus:0.1.1'
  testcompile "org.jetbrains.kotlin:kotlin-test:$kotlin_version"
  testcompile "org.jetbrains.kotlin:kotlin-test-junit:$kotlin_version"
}

mainclassname = "kanvaskt"

generategrammarsource {
    maxheapsize = "64m"
    arguments += ['-package', 'me.tomassetti.lexers']
    outputdirectory = new file("generated-src/antlr/main/me/tomassetti/lexers".tostring())
}
compilejava.dependson generategrammarsource
sourcesets {
    generated {
        java.srcdir 'generated-src/antlr/main/'
    }
}
compilejava.source sourcesets.generated.java, sourcesets.main.java

clean {
    delete "generated-src"
}

idea {
    module {
        sourcedirs += file("generated-src/antlr/main")
    }
}

this should be pretty straightforward. a few comments:

  • our editor will be based on the rsyntaxtextarea component so we added the corresponding dependency
  • we are specifying the version of kotlin we are using. you do not need to install kotlin on your system: gradle will download the compiler and use it based on this configuration
  • we are using antlr to generate the lexer from our lexer grammar
  • we are using the idea plugin: by running ./gradlew idea we can generate an idea project. note that we add the generated code to the list of source directories
  • when we run ./gradlew clean we want also to delete the generated code

changes to the antlr lexer

when using a lexer to feed a parser we can safely ignore some tokens. for example, we could want to ignore spaces and comments. when using a lexer for syntax highlighting instead we want to capture all possible tokens. so we need to slightly adapt the lexer grammar we defined in the first post by:

  • defining a second channel where we will send the tokens which are useless for the parser but needed for syntax highlighting
  • we will add a new real named unmatched to capture all characters which do not belong to correct tokens. this is necessary both because invalid tokens need to be displayed in the editor and because while typing the user would temporarily introduce invalid tokens all the time

the parser will consider only tokens in the default channel, while we will use the default channel and the extra channel in our syntax highlighter.

this is the resulting grammar:

lexer grammar sandylexer;

@lexer::members {
    private static final int extra = 1;
}


// whitespace
newline            : '\r\n' | 'r' | '\n' ;
ws                 : [\t ]+ -> channel(extra) ;

// keywords
var                : 'var' ;

// literals
intlit             : '0'|[1-9][0-9]* ;
declit             : '0'|[1-9][0-9]* '.' [0-9]+ ;

// operators
plus               : '+' ;
minus              : '-' ;
asterisk           : '*' ;
division           : '/' ;
assign             : '=' ;
lparen             : '(' ;
rparen             : ')' ;

// identifiers
id                 : [_]*[a-z][a-za-z0-9_]* ;

unmatched          : .  -> channel(extra);

our editor

our editor will support syntax highlighting for multiple languages. for now, i have implemented support for python and for sandy, the simple language we are working on in this series of posts. for implementing syntax highlighting for python i started from an antlr grammar for python . i started by removing the parser rules, leaving only the lexer rules. then u did very minimal changes to get the final lexer present in the repository.

to create the gui we will need some pretty boring swing code. we basically need to create a frame with a menu. the frame will contained a tabbed panel. we will have one tab per file. in each tab we will have just one editor, implemented using an rsyntaxtextarea.

fun createandshowkanvasgui() {
    uimanager.setlookandfeel(uimanager.getsystemlookandfeelclassname())

    val xtoolkit = toolkit.getdefaulttoolkit()
    val awtappclassnamefield = xtoolkit.javaclass.getdeclaredfield("awtappclassname")
    awtappclassnamefield.isaccessible = true
    awtappclassnamefield.set(xtoolkit, app_title)

    val frame = jframe(app_title)
    frame.background = background_darker
    frame.contentpane.background = background_darker
    frame.defaultcloseoperation = jframe.exit_on_close

    val tabbedpane = mytabbedpane()
    frame.contentpane.add(tabbedpane)

    val menubar = jmenubar()
    val filemenu = jmenu("file")
    menubar.add(filemenu)
    val open = jmenuitem("open")
    open.addactionlistener { opencommand(tabbedpane) }
    filemenu.add(open)
    val new = jmenuitem("new")
    new.addactionlistener { addtab(tabbedpane, "<unnamed>", font) }
    filemenu.add(new)
    val save = jmenuitem("save")
    save.addactionlistener { savecommand(tabbedpane) }
    filemenu.add(save)
    val saveas = jmenuitem("save as")
    saveas.addactionlistener { saveascommand(tabbedpane) }
    filemenu.add(saveas)
    val close = jmenuitem("close")
    close.addactionlistener { closecommand(tabbedpane) }
    filemenu.add(close)
    frame.jmenubar = menubar

    frame.pack()
    if (frame.width < 500) {
        frame.size = dimension(500, 500)
    }
    frame.isvisible = true
}

fun main(args: array<string>) {
    languagesupportregistry.register("py", pythonlanguagesupport)
    languagesupportregistry.register("sandy", sandylanguagesupport)
    swingutilities.invokelater { createandshowkanvasgui() }
}

note that in the main function we register support for our two languages. in the future we could envision a pluggable system to support more languages.

the specific support for sandy is defined in the object sandylanguagesupport (for java developers: an object is just a singleton instance of a class). the support needs a syntaxscheme and an antlrlexerfactory .

the syntaxscheme just returns the style for each given token type. quite easy, eh?

object sandysyntaxscheme : syntaxscheme(true) {
    override fun getstyle(index: int): style {
        val style = style()
        val color = when (index) {
            sandylexer.var -> color.green
            sandylexer.assign -> color.green
            sandylexer.asterisk, sandylexer.division, sandylexer.plus, sandylexer.minus -> color.white
            sandylexer.intlit, sandylexer.declit -> color.blue
            sandylexer.unmatched -> color.red
            sandylexer.id -> color.magenta
            sandylexer.lparen, sandylexer.rparen -> color.white
            else -> null
        }
        if (color != null) {
            style.foreground = color
        }
        return style
    }
}

object sandylanguagesupport : languagesupport {
    override val syntaxscheme: syntaxscheme
        get() = sandysyntaxscheme
    override val antlrlexerfactory: antlrlexerfactory
        get() = object : antlrlexerfactory {
            override fun create(code: string): lexer = sandylexer(org.antlr.v4.runtime.antlrinputstream(code))
        }
}

now, let’s take a look at the antlrlexerfactory . this factory just instantiate an antlr lexer for a certain language. it can be used together with an antlrtokenmaker . the antlrtokenmaker is the adapter between the antlr lexer and the tokenmaker used by the rsyntaxtextarea to process the text. basically a tokenmakerbase is invoked for each line of the file separately as the line is changed. so we ask our antlr lexer to process just the line and we get the resulting tokens and instantiate the tokenimpl instances which are expected by the rsyntaxtextarea.

interface antlrlexerfactory {
    fun create(code: string) : lexer
}

class antlrtokenmaker(val antlrlexerfactory : antlrlexerfactory) : tokenmakerbase() {

    fun tolist(text: segment, startoffset: int, antlrtokens:list<org.antlr.v4.runtime.token>) : token?{
        if (antlrtokens.isempty()) {
            return null
        } else {
            val at = antlrtokens[0]
            val t = tokenimpl(text, text.offset + at.startindex, text.offset + at.startindex + at.text.length - 1, startoffset + at.startindex, at.type, 0)
            t.nexttoken = tolist(text, startoffset, antlrtokens.sublist(1, antlrtokens.size))
            return t
        }
    }

    override fun gettokenlist(text: segment?, initialtokentype: int, startoffset: int): token {
        if (text == null) {
            throw illegalargumentexception()
        }
        val lexer = antlrlexerfactory.create(text.tostring())
        val tokens = linkedlist<org.antlr.v4.runtime.token>()
        while (!lexer._hiteof) {
            tokens.add(lexer.nexttoken())
        }
        return tolist(text, startoffset, tokens) as token
    }

}

this would be a problem for tokens spanning multiple lines. there are workarounds for that but for now let’s keep the things simple and not consider that, given that sandy has no tokens spanning multiple lines.

conclusions

of course this editor is not feature complete. however, i think it is interesting to see how an editor with syntax highlighting can be built with a couple of hundred of lines of quite simple kotlin code.

in future posts we will see how to enrich this editor building things like auto-completion.

Syntax highlighting Syntax (programming languages) ANTLR Kotlin (programming language) code style POST (HTTP)

Published at DZone with permission of Federico Tomassetti, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Event-Driven Architecture Using Serverless Technologies
  • Using Render Log Streams to Log to Papertrail
  • Revolutionizing Algorithmic Trading: The Power of Reinforcement Learning
  • Personalized Code Searches Using OpenGrok

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: