Если по простому - разбивайте строку через regexp.
String text = "5+(2-3*8/10)/123=ERR";
Pattern pattern = Pattern.compile("[+|\\-|\\*|/|=|\\(|\\)]");
Matcher matcher = pattern.matcher(text);
int prev = 0;
while(matcher.find()) {
String operator = text.substring(matcher.start(), matcher.end());
String operand = text.substring(prev, matcher.start());
System.out.printf("Operand: '%s', operator: '%s'\n", operand, operator);
prev = matcher.end();
}
if(prev > 0 && prev < text.length()) {
String last = text.substring(prev, text.length());
System.out.printf("Last: %s\n", last);
}
Operand: '5', operator: '+'
Operand: '', operator: '('
Operand: '2', operator: '-'
Operand: '3', operator: '*'
Operand: '8', operator: '/'
Operand: '10', operator: ')'
Operand: '', operator: '/'
Operand: '123', operator: '='
Last: ERR
Ну а если по взрослому - то вам в antlr -
https://www.antlr.org/ или javaCC -
https://javacc.github.io/javacc/
ЗЫ. для antlr есть фал грамматики для калькулятора -
https://github.com/antlr/grammars-v4/tree/master/c...
И на почитать -
https://www.baeldung.com/java-antlr