Semantic-Guided Temporal Aggregator