图文精华

About云-梭伦科技»专题 › 技术学习(版主发帖区) › 大数据学习 › Mapreduce › 用Mahout构建职位推荐引擎
分享

用Mahout构建职位推荐引擎

52Pig 发表于 2014-11-8 22:40:31 [显示全部楼层] 只看大图 回帖奖励 阅读模式 关闭右栏 2 14225
About云VIP帮助找工作和提升
本帖最后由 52Pig 于 2014-11-8 22:39 编辑
阅读导读：
1.如何设计职位推荐引擎的指标？
2.简述职位推荐引擎所需要的系统架构？
3.如何对推荐结果进行人工比较？
4.职位推荐引擎中什么情况的数据最好做排除？



1. Mahout推荐系统框架概述

Mahout框架包含了一套完整的推荐系统引擎，标准化的数据结构，多样的算法实现，简单的开发流程。Mahout推荐的推荐系统引擎是模块化的，分为5个主要部分组成：数据模型，相似度算法，近邻算法，推荐算法，算法评分器。

更详细的介绍，请参考文章：从源代码剖析Mahout推荐引擎

2. 需求分析：职位推荐引擎指标设计

下面我们将从一个公司案例出发来全面的解释，如何进行职位推荐引擎指标设计。

案例介绍：
互联网某职业社交网站，主要产品包括 个人简历展示页，人脉圈，微博及分享链接，职位发布，职位申请，教育培训等。

用户在完成注册后，需要完善自己的个人信息，包括教育背景，工作经历，项目经历，技能专长等等信息。然后，你要告诉网站，你是否想找工作！！当你选择“是”（求职中），网站会从数据库中为你推荐你可能感兴趣的职位。

通过简短的描述，我们可以粗略地看出，这家职业社交网站的定位和主营业务。核心点有2个：
  • 用户：尽可能多的保存有效完整的用户资料
  • 服务：帮助用户找到工作，帮助猎头和企业找到员工

因此，职位推荐引擎 将成为这个网站的核心功能。
KPI指标设计
  • 通过推荐带来的职位浏览量: 职位网页的PV
  • 通过推荐带来的职位申请量: 职位网页的有效转化

3. 算法模型：推荐算法

2个测试数据集：
  • pv.csv: 职位被浏览的信息,包括用户ID，职位ID
  • job.csv: 职位基本信息,包括职位ID，发布时间，工资标准

1). pv.csv
  • 2列数据：用户ID，职位ID(userid,jobid)
  • 浏览记录:2500条
  • 用户数:1000个，用户ID:1-1000
  • 职位数:200个，职位ID：1-200
部分数据：
  1. 1,11

  2. 2,136

  3. 2,187

  4. 3,165

  5. 3,1

  6. 3,24

  7. 4,8

  8. 4,199

  9. 5,32

  10. 5,100

  11. 6,14

  12. 7,59

  13. 7,147

  14. 8,92

  15. 9,165

  16. 9,80

  17. 9,171

  18. 10,45

  19. 10,31

  20. 10,1

  21. 10,152
复制代码
2). job.csv
  • 3列数据：职位ID，发布时间，工资标准(jobid,create_date,salary)
  • 职位数:200个，职位ID：1-200

部分数据：
  1. 1,2013-01-24,5600

  2. 2,2011-03-02,5400

  3. 3,2011-03-14,8100

  4. 4,2012-10-05,2200

  5. 5,2011-09-03,14100

  6. 6,2011-03-05,6500

  7. 7,2012-06-06,37000

  8. 8,2013-02-18,5500

  9. 9,2010-07-05,7500

  10. 10,2010-01-23,6700

  11. 11,2011-09-19,5200

  12. 12,2010-01-19,29700

  13. 13,2013-09-28,6000

  14. 14,2013-10-23,3300

  15. 15,2010-10-09,2700

  16. 16,2010-07-14,5100

  17. 17,2010-05-13,29000

  18. 18,2010-01-16,21800

  19. 19,2013-05-23,5700

  20. 20,2011-04-24,5900
复制代码
为了完成KPI的指标，我们把问题用“技术”语言转化一下：我们需要让职位的推荐结果更准确，从而增加用户的点击。
  • 1. 组合使用推荐算法，选出“评估推荐器”验证得分较高的算法
  • 2. 人工验证推荐结果
  • 3. 职位有时效性，推荐的结果应该是发布半年内的职位
  • 4. 工资的标准，应不低于用户浏览职位工资的平均值的80%

我们选择UserCF,ItemCF,SlopeOne的 3种推荐算法，进行7种组合的测试。
  • userCF1: LogLikelihoodSimilarity + NearestNUserNeighborhood + GenericBooleanPrefUserBasedRecommender
  • userCF2: CityBlockSimilarity+ NearestNUserNeighborhood + GenericBooleanPrefUserBasedRecommender
  • userCF3: UserTanimoto + NearestNUserNeighborhood + GenericBooleanPrefUserBasedRecommender
  • itemCF1: LogLikelihoodSimilarity + GenericBooleanPrefItemBasedRecommender
  • itemCF2: CityBlockSimilarity+ GenericBooleanPrefItemBasedRecommender
  • itemCF3: ItemTanimoto + GenericBooleanPrefItemBasedRecommender
  • slopeOne：SlopeOneRecommender

关于的推荐算法的详细介绍，请参考文章：Mahout推荐算法API详解

关于算法的组合的详细介绍，请参考文章：从源代码剖析Mahout推荐引擎

4. 架构设计：职位推荐引擎系统架构
mahout-recommend-job-architect.png
上图中，左边是Application业务系统，右边是Mahout，下边是Hadoop集群。
  • 1. 当数据量不太大时，并且算法复杂，直接选择用Mahout读取CSV或者Database数据，在单机内存中进行计算。Mahout是多线程的应用，会并行使用单机所有系统资源。
  • 2. 当数据量很大时，选择并行化算法(ItemCF)，先业务系统的数据导入到Hadoop的HDFS中，然后用Mahout访问HDFS实现算法，这时算法的性能与整个Hadoop集群有关。
  • 3. 计算后的结果，保存到数据库中，方便查询

5. 程序开发：基于Mahout的推荐算法实现

开发环境mahout版本为0.8。 ，请参考文章：用Maven构建Mahout项目

新建Java类：
  • RecommenderEvaluator.java, 选出“评估推荐器”验证得分较高的算法
  • RecommenderResult.java, 对指定数量的结果人工比较
  • RecommenderFilterOutdateResult.java，排除过期职位
  • RecommenderFilterSalaryResult.java，排除工资过低的职位
1). RecommenderEvaluator.java, 选出“评估推荐器”验证得分较高的算
源代码：
  1. public class RecommenderEvaluator {

  2. 

  3.     final static int NEIGHBORHOOD_NUM = 2;

  4.     final static int RECOMMENDER_NUM = 3;

  5. 

  6.     public static void main(String[] args) throws TasteException, IOException {

  7.         String file = "datafile/job/pv.csv";

  8.         DataModel dataModel = RecommendFactory.buildDataModelNoPref(file);

  9.         userLoglikelihood(dataModel);

  10.         userCityBlock(dataModel);

  11.         userTanimoto(dataModel);

  12.         itemLoglikelihood(dataModel);

  13.         itemCityBlock(dataModel);

  14.         itemTanimoto(dataModel);

  15.         slopeOne(dataModel);

  16.     }

  17. 

  18.     public static RecommenderBuilder userLoglikelihood(DataModel dataModel) throws TasteException, IOException {

  19.         System.out.println("userLoglikelihood");

  20.         UserSimilarity userSimilarity = RecommendFactory.userSimilarity(RecommendFactory.SIMILARITY.LOGLIKELIHOOD, dataModel);

  21.         UserNeighborhood userNeighborhood = RecommendFactory.userNeighborhood(RecommendFactory.NEIGHBORHOOD.NEAREST, userSimilarity, dataModel, NEIGHBORHOOD_NUM);

  22.         RecommenderBuilder recommenderBuilder = RecommendFactory.userRecommender(userSimilarity, userNeighborhood, false);

  23. 

  24.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  25.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  26.         return recommenderBuilder;

  27.     }

  28. 

  29.     public static RecommenderBuilder userCityBlock(DataModel dataModel) throws TasteException, IOException {

  30.         System.out.println("userCityBlock");

  31.         UserSimilarity userSimilarity = RecommendFactory.userSimilarity(RecommendFactory.SIMILARITY.CITYBLOCK, dataModel);

  32.         UserNeighborhood userNeighborhood = RecommendFactory.userNeighborhood(RecommendFactory.NEIGHBORHOOD.NEAREST, userSimilarity, dataModel, NEIGHBORHOOD_NUM);

  33.         RecommenderBuilder recommenderBuilder = RecommendFactory.userRecommender(userSimilarity, userNeighborhood, false);

  34. 

  35.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  36.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  37.         return recommenderBuilder;

  38.     }

  39. 

  40.     public static RecommenderBuilder userTanimoto(DataModel dataModel) throws TasteException, IOException {

  41.         System.out.println("userTanimoto");

  42.         UserSimilarity userSimilarity = RecommendFactory.userSimilarity(RecommendFactory.SIMILARITY.TANIMOTO, dataModel);

  43.         UserNeighborhood userNeighborhood = RecommendFactory.userNeighborhood(RecommendFactory.NEIGHBORHOOD.NEAREST, userSimilarity, dataModel, NEIGHBORHOOD_NUM);

  44.         RecommenderBuilder recommenderBuilder = RecommendFactory.userRecommender(userSimilarity, userNeighborhood, false);

  45. 

  46.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  47.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  48.         return recommenderBuilder;

  49.     }

  50. 

  51.     public static RecommenderBuilder itemLoglikelihood(DataModel dataModel) throws TasteException, IOException {

  52.         System.out.println("itemLoglikelihood");

  53.         ItemSimilarity itemSimilarity = RecommendFactory.itemSimilarity(RecommendFactory.SIMILARITY.LOGLIKELIHOOD, dataModel);

  54.         RecommenderBuilder recommenderBuilder = RecommendFactory.itemRecommender(itemSimilarity, false);

  55. 

  56.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  57.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  58.         return recommenderBuilder;

  59.     }

  60. 

  61.     public static RecommenderBuilder itemCityBlock(DataModel dataModel) throws TasteException, IOException {

  62.         System.out.println("itemCityBlock");

  63.         ItemSimilarity itemSimilarity = RecommendFactory.itemSimilarity(RecommendFactory.SIMILARITY.CITYBLOCK, dataModel);

  64.         RecommenderBuilder recommenderBuilder = RecommendFactory.itemRecommender(itemSimilarity, false);

  65. 

  66.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  67.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  68.         return recommenderBuilder;

  69.     }

  70. 

  71.     public static RecommenderBuilder itemTanimoto(DataModel dataModel) throws TasteException, IOException {

  72.         System.out.println("itemTanimoto");

  73.         ItemSimilarity itemSimilarity = RecommendFactory.itemSimilarity(RecommendFactory.SIMILARITY.TANIMOTO, dataModel);

  74.         RecommenderBuilder recommenderBuilder = RecommendFactory.itemRecommender(itemSimilarity, false);

  75. 

  76.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  77.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  78.         return recommenderBuilder;

  79.     }

  80. 

  81.     public static RecommenderBuilder slopeOne(DataModel dataModel) throws TasteException, IOException {

  82.         System.out.println("slopeOne");

  83.         RecommenderBuilder recommenderBuilder = RecommendFactory.slopeOneRecommender();

  84. 

  85.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  86.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  87.         return recommenderBuilder;

  88.     }

  89. 

  90.     public static RecommenderBuilder knnLoglikelihood(DataModel dataModel) throws TasteException, IOException {

  91.         System.out.println("knnLoglikelihood");

  92.         ItemSimilarity itemSimilarity = RecommendFactory.itemSimilarity(RecommendFactory.SIMILARITY.LOGLIKELIHOOD, dataModel);

  93.         RecommenderBuilder recommenderBuilder = RecommendFactory.itemKNNRecommender(itemSimilarity, new NonNegativeQuadraticOptimizer(), 10);

  94. 

  95.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  96.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  97. 

  98.         return recommenderBuilder;

  99.     }

  100. 

  101.     public static RecommenderBuilder knnTanimoto(DataModel dataModel) throws TasteException, IOException {

  102.         System.out.println("knnTanimoto");

  103.         ItemSimilarity itemSimilarity = RecommendFactory.itemSimilarity(RecommendFactory.SIMILARITY.TANIMOTO, dataModel);

  104.         RecommenderBuilder recommenderBuilder = RecommendFactory.itemKNNRecommender(itemSimilarity, new NonNegativeQuadraticOptimizer(), 10);

  105. 

  106.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  107.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  108. 

  109.         return recommenderBuilder;

  110.     }

  111. 

  112.     public static RecommenderBuilder knnCityBlock(DataModel dataModel) throws TasteException, IOException {

  113.         System.out.println("knnCityBlock");

  114.         ItemSimilarity itemSimilarity = RecommendFactory.itemSimilarity(RecommendFactory.SIMILARITY.CITYBLOCK, dataModel);

  115.         RecommenderBuilder recommenderBuilder = RecommendFactory.itemKNNRecommender(itemSimilarity, new NonNegativeQuadraticOptimizer(), 10);

  116. 

  117.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  118.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  119. 

  120.         return recommenderBuilder;

  121.     }

  122. 

  123.     public static RecommenderBuilder svd(DataModel dataModel) throws TasteException {

  124.         System.out.println("svd");

  125.         RecommenderBuilder recommenderBuilder = RecommendFactory.svdRecommender(new ALSWRFactorizer(dataModel, 5, 0.05, 10));

  126. 

  127.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  128.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  129. 

  130.         return recommenderBuilder;

  131.     }

  132. 

  133.     public static RecommenderBuilder treeClusterLoglikelihood(DataModel dataModel) throws TasteException {

  134.         System.out.println("treeClusterLoglikelihood");

  135.         UserSimilarity userSimilarity = RecommendFactory.userSimilarity(RecommendFactory.SIMILARITY.LOGLIKELIHOOD, dataModel);

  136.         ClusterSimilarity clusterSimilarity = RecommendFactory.clusterSimilarity(RecommendFactory.SIMILARITY.FARTHEST_NEIGHBOR_CLUSTER, userSimilarity);

  137.         RecommenderBuilder recommenderBuilder = RecommendFactory.treeClusterRecommender(clusterSimilarity, 3);

  138. 

  139.         RecommendFactory.evaluate(RecommendFactory.EVALUATOR.AVERAGE_ABSOLUTE_DIFFERENCE, recommenderBuilder, null, dataModel, 0.7);

  140.         RecommendFactory.statsEvaluator(recommenderBuilder, null, dataModel, 2);

  141. 

  142.         return recommenderBuilder;

  143.     }

  144. }
复制代码
运行结果，控制台输出：
  1. userLoglikelihood

  2. AVERAGE_ABSOLUTE_DIFFERENCE Evaluater Score:0.2741487771272658

  3. Recommender IR Evaluator: [Precision:0.6424242424242422,Recall:0.4098360655737705]

  4. userCityBlock

  5. AVERAGE_ABSOLUTE_DIFFERENCE Evaluater Score:0.575306732961736

  6. Recommender IR Evaluator: [Precision:0.919580419580419,Recall:0.4371584699453552]

  7. userTanimoto

  8. AVERAGE_ABSOLUTE_DIFFERENCE Evaluater Score:0.5546485136181523

  9. Recommender IR Evaluator: [Precision:0.6625766871165644,Recall:0.41803278688524603]

  10. itemLoglikelihood

  11. AVERAGE_ABSOLUTE_DIFFERENCE Evaluater Score:0.5398332608612343

  12. Recommender IR Evaluator: [Precision:0.26229508196721296,Recall:0.26229508196721296]

  13. itemCityBlock

  14. AVERAGE_ABSOLUTE_DIFFERENCE Evaluater Score:0.9251437840891661

  15. Recommender IR Evaluator: [Precision:0.02185792349726776,Recall:0.02185792349726776]

  16. itemTanimoto

  17. AVERAGE_ABSOLUTE_DIFFERENCE Evaluater Score:0.9176432856689655

  18. Recommender IR Evaluator: [Precision:0.26229508196721296,Recall:0.26229508196721296]

  19. slopeOne

  20. AVERAGE_ABSOLUTE_DIFFERENCE Evaluater Score:0.0

  21. Recommender IR Evaluator: [Precision:0.01912568306010929,Recall:0.01912568306010929]
复制代码
可视化“评估推荐器”输出：
difference.png
evaluator.png
UserCityBlock算法评估的结果是最好的，基于UserCF的算法比ItemCF都要好，SlopeOne算法几乎没有得分。

2). RecommenderResult.java, 对指定数量的结果人工比较
为得到差异化结果，我们分别取UserCityBlock,itemLoglikelihood，对推荐结果人工比较。

源代码：
  1. public class RecommenderResult {

  2. 

  3.     final static int NEIGHBORHOOD_NUM = 2;

  4.     final static int RECOMMENDER_NUM = 3;

  5. 

  6.     public static void main(String[] args) throws TasteException, IOException {

  7.         String file = "datafile/job/pv.csv";

  8.         DataModel dataModel = RecommendFactory.buildDataModelNoPref(file);

  9.         RecommenderBuilder rb1 = RecommenderEvaluator.userCityBlock(dataModel);

  10.         RecommenderBuilder rb2 = RecommenderEvaluator.itemLoglikelihood(dataModel);

  11. 

  12.         LongPrimitiveIterator iter = dataModel.getUserIDs();

  13.         while (iter.hasNext()) {

  14.             long uid = iter.nextLong();

  15.             System.out.print("userCityBlock    =>");

  16.             result(uid, rb1, dataModel);

  17.             System.out.print("itemLoglikelihood=>");

  18.             result(uid, rb2, dataModel);

  19.         }

  20.     }

  21. 

  22.     public static void result(long uid, RecommenderBuilder recommenderBuilder, DataModel dataModel) throws TasteException {

  23.         List list = recommenderBuilder.buildRecommender(dataModel).recommend(uid, RECOMMENDER_NUM);

  24.         RecommendFactory.showItems(uid, list, false);

  25.     }

  26. 

  27. }
复制代码
控制台输出：只截取部分结果
  1. ...

  2. userCityBlock    =>uid:968,(61,0.333333)

  3. itemLoglikelihood=>uid:968,(121,1.429362)(153,1.239939)(198,1.207726)

  4. userCityBlock    =>uid:969,

  5. itemLoglikelihood=>uid:969,(75,1.326499)(30,0.873100)(85,0.763344)

  6. userCityBlock    =>uid:970,

  7. itemLoglikelihood=>uid:970,(13,0.748417)(156,0.748417)(122,0.748417)

  8. userCityBlock    =>uid:971,

  9. itemLoglikelihood=>uid:971,(38,2.060951)(104,1.951208)(83,1.941735)

  10. userCityBlock    =>uid:972,

  11. itemLoglikelihood=>uid:972,(131,1.378395)(4,1.349386)(87,0.881816)

  12. userCityBlock    =>uid:973,

  13. itemLoglikelihood=>uid:973,(196,1.432040)(140,1.398066)(130,1.380335)

  14. userCityBlock    =>uid:974,(19,0.200000)

  15. itemLoglikelihood=>uid:974,(145,1.994049)(121,1.794289)(98,1.738027)

  16. ...
复制代码
我们查看uid=974的用户推荐信息：
搜索pv.csv：
  1. > pv[which(pv$userid==974),]

  2.      userid jobid

  3. 2426    974   106

  4. 2427    974   173

  5. 2428    974    82

  6. 2429    974   188

  7. 2430    974    78
复制代码
搜索job.csv:
  1. > job[job$jobid %in% c(145,121,98,19),]

  2.     jobid create_date salary

  3. 19     19  2013-05-23   5700

  4. 98     98  2010-01-15   2900

  5. 121   121  2010-06-19   5300

  6. 145   145  2013-08-02   6800
复制代码
上面两种算法，推荐的结果都是2010年的职位，这些结果并不是太好，接下来我们要排除过期职位，只保留2013年的职位。
3).RecommenderFilterOutdateResult.java，排除过期职位
源代码：
  1. public class RecommenderFilterOutdateResult {

  2. 

  3.     final static int NEIGHBORHOOD_NUM = 2;

  4.     final static int RECOMMENDER_NUM = 3;

  5. 

  6.     public static void main(String[] args) throws TasteException, IOException {

  7.         String file = "datafile/job/pv.csv";

  8.         DataModel dataModel = RecommendFactory.buildDataModelNoPref(file);

  9.         RecommenderBuilder rb1 = RecommenderEvaluator.userCityBlock(dataModel);

  10.         RecommenderBuilder rb2 = RecommenderEvaluator.itemLoglikelihood(dataModel);

  11. 

  12.         LongPrimitiveIterator iter = dataModel.getUserIDs();

  13.         while (iter.hasNext()) {

  14.             long uid = iter.nextLong();

  15.             System.out.print("userCityBlock    =>");

  16.             filterOutdate(uid, rb1, dataModel);

  17.             System.out.print("itemLoglikelihood=>");

  18.             filterOutdate(uid, rb2, dataModel);

  19.         }

  20.     }

  21. 

  22.     public static void filterOutdate(long uid, RecommenderBuilder recommenderBuilder, DataModel dataModel) throws TasteException, IOException {

  23.         Set jobids = getOutdateJobID("datafile/job/job.csv");

  24.         IDRescorer rescorer = new JobRescorer(jobids);

  25.         List list = recommenderBuilder.buildRecommender(dataModel).recommend(uid, RECOMMENDER_NUM, rescorer);

  26.         RecommendFactory.showItems(uid, list, true);

  27.     }

  28. 

  29.     public static Set getOutdateJobID(String file) throws IOException {

  30.         BufferedReader br = new BufferedReader(new FileReader(new File(file)));

  31.         Set jobids = new HashSet();

  32.         String s = null;

  33.         while ((s = br.readLine()) != null) {

  34.             String[] cols = s.split(",");

  35.             SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd");

  36.             Date date = null;

  37.             try {

  38.                 date = df.parse(cols[1]);

  39.                 if (date.getTime() < df.parse("2013-01-01").getTime()) {

  40.                     jobids.add(Long.parseLong(cols[0]));

  41.                 }

  42.             } catch (ParseException e) {

  43.                 e.printStackTrace();

  44.             }

  45. 

  46.         }

  47.         br.close();

  48.         return jobids;

  49.     }

  50. 

  51. }

  52. 

  53. class JobRescorer implements IDRescorer {

  54.     final private Set jobids;

  55. 

  56.     public JobRescorer(Set jobs) {

  57.         this.jobids = jobs;

  58.     }

  59. 

  60.     @Override

  61.     public double rescore(long id, double originalScore) {

  62.         return isFiltered(id) ? Double.NaN : originalScore;

  63.     }

  64. 

  65.     @Override

  66.     public boolean isFiltered(long id) {

  67.         return jobids.contains(id);

  68.     }

  69. }
复制代码
控制台输出：只截取部分结果
  1. ...

  2. itemLoglikelihood=>uid:965,(200,0.829600)(122,0.748417)(170,0.736340)

  3. userCityBlock    =>uid:966,(114,0.250000)

  4. itemLoglikelihood=>uid:966,(114,1.516898)(101,0.864536)(99,0.856057)

  5. userCityBlock    =>uid:967,

  6. itemLoglikelihood=>uid:967,(105,0.873100)(114,0.725016)(168,0.707119)

  7. userCityBlock    =>uid:968,

  8. itemLoglikelihood=>uid:968,(174,0.735004)(39,0.696716)(185,0.696171)

  9. userCityBlock    =>uid:969,

  10. itemLoglikelihood=>uid:969,(197,0.723203)(81,0.710230)(167,0.668358)

  11. userCityBlock    =>uid:970,

  12. itemLoglikelihood=>uid:970,(13,0.748417)(122,0.748417)(28,0.736340)

  13. userCityBlock    =>uid:971,

  14. itemLoglikelihood=>uid:971,(28,1.540753)(174,1.511881)(39,1.435575)

  15. userCityBlock    =>uid:972,

  16. itemLoglikelihood=>uid:972,(14,0.800605)(60,0.794088)(163,0.710230)

  17. userCityBlock    =>uid:973,

  18. itemLoglikelihood=>uid:973,(56,0.795529)(13,0.712680)(120,0.701026)

  19. userCityBlock    =>uid:974,(19,0.200000)

  20. itemLoglikelihood=>uid:974,(145,1.994049)(89,1.578694)(19,1.435193)

  21. ...
复制代码
我们查看uid=994的用户推荐信息：
搜索pv.csv：
  1. > pv[which(pv$userid==974),]

  2.      userid jobid

  3. 2426    974   106

  4. 2427    974   173

  5. 2428    974    82

  6. 2429    974   188

  7. 2430    974    78
复制代码
搜索job.csv:
  1. > job[job$jobid %in% c(19,145,89),]

  2.     jobid create_date salary

  3. 19     19  2013-05-23   5700

  4. 89     89  2013-06-15   8400

  5. 145   145  2013-08-02   6800
复制代码
排除过期的职位比较，我们发现userCityBlock结果都是19，itemLoglikelihood的第2，3的结果被替换为了得分更低的89和19。
4).RecommenderFilterSalaryResult.java，排除工资过低的职位
我们查看uid=994的用户，浏览过的职位。
  1. > job[job$jobid %in% c(106,173,82,188,78),]

  2.     jobid create_date salary

  3. 78     78  2012-01-29   6800

  4. 82     82  2010-07-05   7500

  5. 106   106  2011-04-25   5200

  6. 173   173  2013-09-13   5200

  7. 188   188  2010-07-14   6000
复制代码
平均工资为=6140，我们觉得用户的浏览职位的行为，一般不会看比自己现在工资低的职位，因此设计算法，排除工资低于平均工资80%的职位，即排除工资小于4912的推荐职位(6140*0.8=4912)

大家可以参考上文中RecommenderFilterOutdateResult.java,自行实现。

这样，我们就完成用Mahout构建职位推荐引擎的算法。如果没有Mahout，我们自己写这个算法引擎估计还要花个小半年的时间，善加利用开源技术会帮助我们飞一样的成长！！





本帖被以下淘专辑推荐:

回复

使用道具 举报

自定义广告语

已有(2)人评论

跳转到指定楼层
正序浏览
anyhuayong 发表于 2014-11-10 08:41:17
好资源，收藏了，楼主辛苦
回复

使用道具 举报

net211211 发表于 2015-3-18 11:19:09
好东西。学些一下。
回复

使用道具 举报

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
52Pig

中级会员

关注

  • 49

    主题

  • 57

    帖子

  • 16

    粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000